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ABSTRACT 


The  Navy  is  considering  the  use  of  unmanned  surface  vehicles  (USVs)  to  reduce 
risk  to  personnel  in  maritime  interdiction  operations,  and  to  conduct  intelligence, 
surveillance  and  reconnaissance  (ISR)  and  force  protection  (FP)  missions.  In  this  thesis, 
alternative  configurations  of  the  prototype  and  operational  uses  of  the  USV  are  explored 
using  agent-based  simulation  for  three  scenarios.  An  efficient  experiment  design  alters 
settings  of  ten  factors  for  the  two  ISR  scenarios  and  1 1  factors  for  the  FP  scenario.  Some 
factors  varied  in  the  experiment  are  uncontrollable  during  operations,  such  as  the  total 
number  of  contacts,  threat  density,  their  maneuvering  characteristics,  and  the  sea  state. 
The  USV  sensor  range  and  endurance  are  also  considered  as  well  as  factors  set  by  the 
decision-maker  for  a  particular  mission;  namely,  USV  speed  and  numbers  to  deploy. 
The  results  provide  several  operational  and  tactical  insights  with  implications  for 
patrolling  and  combat  radius,  and  form  the  basis  for  a  recommendation  to  use  the  USV  in 
an  active  role  in  maritime  missions.  The  results  also  support  the  guidance  on  the  benefits 
of  improving  USV  sensing  and  endurance  capabilities,  and  find  that  simply  increasing 
USV  numbers  is  not  necessary  for  attaining  high  mission  performance. 
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EXECUTIVE  SUMMARY 

The  Navy  is  considering  distributed  means  of  conducting  surveillance  and 
reconnaissance  using  unmanned  surface  vehicles  (USVs)  (Ricci,  2002).  An  attack  on  24 
April  2004  against  Sailors  in  a  Rigid  Hull  Inflatable  Boat  (RHIB)  makes  the  notion  more 
relevant.  During  maritime  interdiction  operations  (MIO)  in  the  Arabian  Gulf,  a  7- 
member  crew  RHIB  proceeded  to  intercept  and  board  an  unidentified  dhow  for 
investigation.  As  the  RHIB  approached  the  dhow,  it  exploded — skilling  two  Sailors  and 
wounding  four  others.  Two  other  unidentified  dhows  also  exploded  the  same  day  (Navy 
Newsstand,  2004).  These  incidents  give  solid  motivation  for  the  USV  to  be  integrated 
into  daily  operations  in  the  Fleet. 

Using  agent-based  simulation  to  analyze  information,  surveillance,  and 
reconnaissance  (ISR)  and  force  protection  (FP)  missions,  each  model  depicts  USVs, 
enemies,  neutrals,  and  a  high  value  unit  (HVU).  The  USVs  leave  the  HVU  in  pursuit  of 
accurate  identification  of  contacts  in  its  field  of  view.  There  are  two  ISR  models:  a 
Waypoint  scenario  which  provides  a  predetermined  path  for  each  USV  to  follow,  and  an 
Interceptor  scenario,  in  which  the  USV  is  free  to  move  on  any  path.  The  FP  Model  has 
two  types  of  enemies:  threatening  and  non-threatening.  This  model  assesses  the  ability 
of  each  USV  to  prevent  threatening  enemies  from  reaching  the  HVU. 

This  thesis  looks  at  ten  factors  for  the  ISR  Models: 

•  USV  speed, 

•  Enemy  speed, 

•  Neutral  speed, 

•  Sea  state, 

•  Number  of  USVs, 

•  Number  of  Contacts, 

•  Percentage  of  contacts  that  are  enemy, 

•  Sensor  range, 

XV 


•  Tactical  radius  from  the  HVU,  and 

•  Time  on  station. 

The  FP  model  enables  eonsideration  of  eleven  faetors,  ineluding  all  of  those  in  the 
ISR  analysis  with  the  exeeption  of  the  time  on  station  and  with  the  addition  of  threatening 
enemy  speed  and  pereentage  of  enemies  that  are  threatening.  The  measures  of 
effectiveness  (MOEs)  are  the  proportion  of  enemies  detected.  The  FP  seenario  has  two 
additional  MOEs:  the  proportion  of  threatening  enemies  deteeted  and  the  number  of 
threatening  enemies  that  reaeh  the  HVEl. 

One  factor  that  is  significant  in  each  of  the  five  analyses  is  the  number  of  ElSVs. 
USV  speed  is  signifieant  in  all  analyses  except  the  FP-number  that  reaeh  the  HVEl. 
Sensor  range  and  time  on  station  are  important  in  the  two  ISR  seenarios.  Finally,  the 
percentage  of  threatening  enemies  is  signifieant  in  eaeh  of  the  FP  analyses.  EJSV  speed, 
the  number  of  US  Vs  available  to  the  HVU,  sensor  range,  and  time  on  station  are  all 
eontrollable  faetors.  Pereentage  of  threatening  enemies  is  the  only  eommon  factor  that  is 
not  controllable.  This  analysis  shows  that  the  Navy  ean  make  a  deeision  to  deploy  the 
USV  in  one  of  the  three  proposed  seenarios  without  having  to  rely  on  intelligenee  or 
make  assumptions  regarding  inaeeessible  information.  This  is  not  to  say  that  the  models’ 
other  significant  factors  are  trivial;  only  that  if,  for  example,  the  situation  at  hand  would 
evolve  from  an  Intereeptor  seenario  to  a  FP  seenario,  some  important  information  is 
already  known  about  the  impaet  of  the  number  of  USVs,  the  USV  speed,  and  the  sensor 
range. 

Preventing  fatal  ineidents  sueh  as  the  lethal  April  2004  event  is  an  advantage  to 
implementation  of  the  USV  into  the  Fleet.  Multiple  linear  regression  and  regression  trees 
are  coupled  with  an  experimental  design  that  analyzes  up  to  11  factors  simultaneously. 
This  provides  insights  into  USV  eonfiguration  into  the  Fleet  putting  a  stop  to  fatal  MIO 
operations.  These  insights  inelude  working  toward  eovering  a  1600  sq-nm  area  with  16- 
20  USVs  per  HVU,  enabling  the  platforms  to  be  able  to  stay  away  from  the  HVU  for  at 
least  7.5  hours,  and  designing  either  an  inereased  range  that  the  USVs  ean  travel  from  the 
HVU  or  an  improved  sensor  range  to  inerease  the  proportion  of  detections. 
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I.  INTRODUCTION 


A,  UNMANNED  SURFACE  VEHICLES 

The  Navy  is  considering  distributed  means  of  conducting  surveillance  and 
reconnaissance  using  unmanned  srrrface  vehicles  (USVs)  (Ricci,  2002).  An  attack  on  24 
April  2004  against  Sailors  in  a  Rigid  Hull  Inflatable  Boat  (RHIB)  makes  the  notion  more 
relevant.  During  maritime  interdiction  operations  (MIO)  in  the  Arabian  Gulf,  a  7- 
member  crew  RHIB  proceeded  to  intercept  and  board  an  unidentified  dhow  for 
investigation.  As  the  RHIB  approached,  the  dhow  exploded — skilling  two  Sailors  and 
wounding  four  others.  Two  other  unidentified  dhows  also  exploded  the  same  day  (Navy 
Newsstand,  2004).  These  incidents  give  solid  motivation  for  the  USV  to  be  integrated 
with  daily  operations  in  the  Fleet. 

The  US  Navy  has  a  prototype  USV  that  deployed  with  the  USS  GETTYSBURG 
(GET).  Essentially  a  7-meter  RHIB  that  has  been  configured  for  ISR,  the  current  USV 
contains  an  electro-optical/infrared  (EO/IR)  camera,  commercial  grade  radar,  microphone 
and  a  loudspeaker.  It  is  radio  controlled  with  a  current  range  of  five  nautical  miles  (nm) 
from  the  host  ship.  The  USV  is  gas-powered  with  a  projected  endurance  of  six  hours  and 
a  10-foot  height  of  eye.  A  picture  of  Spartan  Scout,  the  prototype  USV,  is  provided  in 
Figure  1. 


Figure  1.  Spartan  Scout  Controlled  from  GET  (Rich,  2003) 
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The  US  Navy  is  in  the  initial  stages  of  procurement  of  the  USV.  Several  potential 
uses  exist.  Surveillance  enables  the  host  ship  to  detect  and  identify  other  objects  on  the 
seas  that  are  outside  of  the  visual  and  radar  range  of  the  vessel  from  which  the  USV  is 
operating.  Along  with  surveillance,  interception,  defined  as  the  ability  to  move  towards 
the  potential  threatening  contact,  is  a  mission  essential  task  especially  for  MIO.  The 
combination  of  surveillance  and  maritime  interdiction  capabilities  expected  from  the 
USV  is  integral  to  provide  the  Navy  the  ability  to  perform  these  missions  while  the  host 
ship  continues  on  operational  tasking  and  maintains  its  position.  Another  need  for  the 
USV  is  Force  Protection  (FP),  as  evidenced  by  the  April  2004  attack  cited  earlier.  The 
host  platform  can  allocate  its  resources  in  different  ways  to  ensure  proper  defense.  Mine 
warfare  is  another  projected  use  of  the  USV,  but  not  covered  in  the  current  study. 

The  operations  of  the  prototype,  Spartan  Scout,  tested  its  intelligence,  surveillance 
and  reconnaissance  (ISR)  capabilities.  The  tests  occurred  on  December  1-2,  2003  and 
January  19-22,  2004.  (Rich,  2003  and  Quarderer,  2004).  Along  with  ISR  information, 
the  other  data  collected  during  this  live  testing  inform  this  current  research  in  determining 
the  benefits  and  shortcomings  of  adding  the  USV  to  missions  in  the  fleet.  Unfortunately, 
the  possibilities  for  gaining  insights  are  limited  when  only  a  single  prototype  is  available. 
Instead,  this  thesis  uses  agent-based  simulation  to  determine  configurations  for  the  USV 
and  the  unit  from  which  the  USV  is  deployed. 

An  agent-based  simulation  is  used  to  evaluate  the  performance  of  configurations 
and  operational  use  of  the  USV.  The  simulation  varies  these  current  characteristics  of  the 
prototype  among  the  missions  expected  of  a  USV  in  the  Navy.  The  results  form  the  basis 
for  a  recommendation  to  the  US  Navy  to  use  the  USV  in  an  active  role  in  maritime 
missions.  The  simulation  looks  at  the  type  of  mission  as  well  as  the  sea  state  in  which  the 
mission  is  to  be  performed  in.  In  order  to  fully  capture  the  essence  of  agent-based 
simulation,  the  model  experiments  with  the  number  of  USV’s  to  be  deployed  per  High 
Value  Unit  (HVU)  throughout  simulations  so  that  activity  differences  can  be  detected 
with  low  and  high  numbers  of  USV’s. 
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B,  PURPOSE  AND  MOTIVATION 

The  Navy  has  only  recently  begun  to  procure  these  assets,  and  it  has  not  yet 
developed  operational  procedures  for  the  USV(s).  Determining  whether  or  not  the  USV 
benefits  fleet  operations  is  desirable.  Being  able  to  emulate  actual  scenarios  that  would 
be  useful  to  the  ships  in  the  fleet  in  an  agent-based  simulation  is  an  objective  of  this  study 
as  well.  Three  scenarios  of  interest  are: 

•  Maintaining  a  recognized  maritime  picture  (RMP)  of  a  large  number  of 

vessels; 

•  Sorting  out  and  tracking  a  specific  contact  of  interest  out  of  a  number  of 
routine  vessels;  and, 

•  Detecting,  identifying  and  tracking  a  high-density  group  of  contacts  of  interest 
among  a  number  of  contacts. 

Another  intention  is  to  estimate  USV  performance  under  a  variety  of  situations 
with  a  confidence  that  is  acceptable  to  the  Navy.  These  performance  estimates  provide 
information  and  insights  that  can  assist  decision-makers  or  lead  to  further  research 
involving  specific  areas  of  interest,  tactical  applications,  or  operational  scenarios. 

Undertaking  this  topic  came  as  a  function  of  the  author’s  future  as  a  Surface 
Warfare  Officer  in  the  Navy.  Since  the  USV  is  in  its  beginning  stages  of  development 
and  testing,  it  appears  to  be  a  great  place  to  begin  research  for  a  thesis  as  well  as 
background  for  a  future  SWO.  Knowing  that  this  thesis  has  the  potential  for  further 
developments  within  the  fleet  or  even  for  further  research  is  inspiring  and  encouraging. 

C.  SCOPE  AND  METHODOLOGY 

This  study  uses  an  agent-based  simulating  platform  PYTHAGORAS  to  model  the 

performance  of  the  USV  with  respect  to  its  current  capabilities.  Agent-based  simulations 

are  those  in  which  the  entities  and  objects  that  make  up  the  model  behave  disjointedly 

(Sanchez  and  Lucas,  2002).  Each  entity  of  a  squad,  for  example,  is  defined  in  the  exact 

same  manner.  The  entities,  or  agents,  act  independently  of  other  agents  in  the  squad.  For 

military  applications,  this  logic  seems  applicable.  Training  in  the  military  is,  for  all 

intents  and  purposes  a  constant  factor  for  the  members,  but  each  member  takes  what  is 

commanded  and,  in  conjunction  with  the  environment,  makes  decisions  separately  from 

the  other  members  in  the  group.  More  detailed  explanations  of  agent-based  simulation 
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and  analysis  can  be  found  in  Sanchez  and  Lucas  (2002).  The  models  developed  in  this 
thesis  are  able  to  capture  the  way  USVs  act  under  a  variety  of  circumstances.  Factors 
that  are  varied  throughout  the  modeling  include; 

•  Sea  state, 

•  Speed  of  USV  and  targets, 

•  How  close  the  USV  must  get  to  a  target  for  accurate  identification, 

•  Number  of  USVs  to  send  out  for  particular  mission,  and 

•  Combat  radius  (the  length  of  time  to  and  on  station)  under  the  various  factors. 

Among  the  factors  that  are  varied,  experiments  determine  with  statistical 

significance  whether  the  manner  of  deploying  USVs  should  be  on  a  given  patrol  pattern 
as  opposed  to  the  USV  choosing  the  closest  enemy  to  pursue.  Another  desired  outcome 
of  this  thesis  is  to  see  if  the  factors  examined  yield  evidence  whether  the  USV  is  the  best 
solution  to  the  tactical  problem.  Varying  these  factors,  in  conjunction  with  operational 
scenarios,  covers  some  ground  to  provide  useful  insights  to  the  Navy,  but  it  is  optimistic 
to  expect  this  study  to  enable  the  necessary  decisions  for  full  implementation  of  the  USV 
into  the  Fleet  under  all  circumstances. 

Every  problem  needs  answers.  Therefore  it  is  necessary  to  define  the  correct 
questions  and  specific  problem  statements  to  be  answered.  The  organization  of  the 
remainder  of  this  encompasses  an  approach  toward  answering  each  of  the  following 
questions  and  statements.  Chapter  II  contains  an  overview  of  the  assumptions  made  to 
develop  the  models,  the  models’  capabilities,  and  descriptions.  Chapter  III  is  the  design 
of  experiments  explaining  the  design  process,  the  factors  analyzed  throughout  analysis, 
and  the  tactical  interpretation.  The  analysis  of  each  model,  verification  and  validation, 
and  results  are  included  in  Chapter  IV.  The  final  chapter  consists  of  the  conclusions  of 
the  analysis,  lessons  learned  regarding  the  use  of  agent-based  simulation  for  analyzing 
USV  deployment  and  recommendations  for  further  research  on  this  topic. 

D.  PAYOFFS  AND  BENEFITS 

This  thesis  benefits  the  researchers  and  supporters  of  the  USV,  and  it  seeks  to 
support  Fleet-wide  decisions  on  whether  the  USV  should  be  implemented  into  tactical 
operations.  One  series  of  experiments  in  this  study  could  aid  in  determining  whether  the 
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current  configuration  of  the  USV  should  be  altered,  including  whether  weapons  should  be 
added  or  if  any  of  the  other  four  capabilities,  (force  protection,  surveillance,  maritime 
interdiction  or  mine  warfare)  should  be  fully  implemented  to  obtain  optimal  performance. 
As  a  direct  link  to  disseminate  information  for  the  benefit  of  USV  researchers  and 
supporters,  the  results  of  this  thesis  are  implemented  into  a  TACMEMO  (Statement  of 
Work,  2003).  The  best  benefit  that  the  Fleet  can  have  is  preventing  the  death  and  injury 
of  Sailors.  The  MIO  incident  on  24  April  2004  is  an  example  of  why  US  Vs  are  necessary 
in  the  current  tactical  environment. 
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II.  SCENARIO  DESCRIPTION 


A.  ASSUMPTIONS  AND  CAPABILITIES 

The  scenarios  included  in  this  study  are  the  prototype  of  the  Spartan  Scout 
(Scenario-P),  two  proposed  Information,  Surveillance  and  Reconnaissance  (ISR) 
scenarios  and  one  proposed  Force  Protection  (FP)  scenario.  The  ISR  models  include  a 
Waypoint  scenario  (Scenario-W)  and  an  Interceptor  scenario  (Scenario-I).  The  proposed 
scenarios  are  explained  in  more  detail  later.  Analyzing  a  tactical  problem  using 
simulation  requires  abstraction  of  a  scenario  using  tactical  information  while  seeking  to 
retain  a  sufficient  level  of  resolution.  For  this  thesis,  several  basic  assumptions  enable 
the  problems  to  be  implemented  on  the  PYTHAGORAS  simulation  platform  and  make 
the  results  comprehensible.  Because  this  is  an  abstraction  of  a  tactical  scenario,  the 
model  needs  to  be  verified  and  validated  so  that  the  results  are  credible  (Law  and  Kelton, 
2000).  The  verification  and  validation  are  expanded  in  Chapter  IV. 

An  important  concept  underlying  all  three  scenarios  is  that  US  Vs  deploy  from 
High  Value  Units  (HVUs).  The  HVUs  can  be  a  Carrier  or  Expeditionary  Strike  Group, 
(CSG  and  ESG,  respectively),  or  any  generic  HVU.  The  HVU  is  composed  of  a  number 
of  ships,  or  even  a  single  defended  asset,  responsible  for  the  each  USV.  It  is  not  correct 
to  assume  that  the  HVU  has  direct  control  of  each  USV,  but  that  individual  units  within 
the  CSG  or  ESG  are  responsible  for  their  respective  USVs.  Eor  simplicity,  the  USVs  in 
the  ISR  and  EP  scenarios  are  not  represented  as  deploying  from  individual  ships.  Instead 
the  group  controls  the  USVs.  When  viewed  within  the  context  of  a  tactical  situation,  ISR 
assets  are  typically  treated  as  common  resources  for  the  entire  task  group.  Therefore,  the 
assumption  that  each  USV  is  controlled  by  any  unit  is  realistic,  and  emulates  how  the 
chain  of  command  may  evolve. 

Another  abstraction  of  the  PYTHAGORAS  models  is  that  some  meteorological 
factors  are  not  included  in  the  development  of  the  simulations,  including  wind,  current, 
tides,  and  sea  surface  temperature.  These  are  factors  that  the  meteorology  and 
oceanographic  (METOC)  community  predict  would  have  an  effect  on  a  USV  (Joint 
METOC  Handbook,  2000).  While  these  factors  are  omitted  from  the  simulation  for 
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simplicity,  their  effects  on  visibility  and  maneuverability  do  exist.  Sea  state  is  another 
factor  important  by  the  METOC  community  and  it  is  included  in  the  simulation.  This  is 
discussed  in  more  detail  in  Chapter  III. 

One  final  faetor  not  ineorporated  into  the  simulations  is  latency.  Latency  is  the 
delay  as  data  are  relayed  from  each  USV  to  the  HVU  so  the  operator  ean  control  the 
vehicle  or  identify  a  contact.  The  average  latency  from  live  USV  prototype 
experimentation  was  0.204  seeonds  with  a  standard  deviation  of  0.037  (Quarderer,  2004). 
Sinee  each  simulation  time  step  is  72  seeonds,  this  small  amount  of  time  for  data  lateney 
essentially  beeomes  absorbed  within  the  time  step.  Therefore  the  simulation  models  are 
slightly  optimistic  because  the  data  latency  does  affeet  control  of  the  USV. 
PYTHAGORAS  ean  model  these  effects  by  increasing  the  “hold  fire  desire”  property  in 
the  “engagement  desires”  in  the  simulation.  “Hold  fire  desire”  can  be  thought  of  as  a 
probability  that  the  agent  has  to  wait  to  act.  This  defines  whether  each  USV  takes  action 
or  not  with  a  certain  non-zero  probability.  An  area  for  further  researeh  is  rescaling  the 
problem  so  that  the  time  steps  are  less  than  one  seeond,  enabling  effective  modeling  of 
latency. 


B,  SCENARIOS 

Scenario-P  closely  represents  the  results  from  several  prototype  exereises 
eondueted  onboard  the  USS  GETTYSBURG  (GET).  The  model  is  based  on  a  single 
USV  operating  within  a  five  nautical  mile  (nm)  radius,  limited  by  the  eontrolling  Radio 
Erequency  (RE)  from  its  host  ship.  The  range  drives  the  overall  dimension  of  the 
seenario,  whieh  is  10  nm  by  10  nm.  The  USV  has  radar  with  a  range  of  16  nm  and  an 
Electro-Optical/Infra-Red  (EO/IR)  eamera  for  visual  detection,  and  sends  its  data  to  the 
host  via  a  real-time  link  to  shipboard  video  eonsoles.  The  USV  pursues  contacts  that 
require  eloser  investigation.  The  eontacts  are  neutral  merchant  ships  and  contacts  of 
interest.  Seenario-P  serves  as  the  base  scenario  for  verification  that  the  simulation 
models  are  in  compliance  with  the  tactical  situation.  The  only  run  done  on  this  scenario 
is  eonfirmatory,  to  verify  its  eoneurrence  with  the  live  prototype  results. 
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The  three  experimental  seenarios  are  designed  with  the  purpose  of  emulating 
operational  possibilities  for  the  USV.  The  scope  of  the  proposed  scenarios  is  an  area  of 
interest  of  1600  square-nm  in  the  open  seas  near  possibly  high-traffic  regions.  Scenarios- 
W  and  -I  are  similar  except  in  one  way — the  USV  movement  patterns.  In  Scenario-W, 
each  USV  has  predetermined  patrol  patterns,  or  waypoints.  Each  USV  is  initially  placed 
at  the  HVU  and  patrols  along  the  prespecified  paths.  Scenario-I  explores  ISR  operations 
when  each  USV  takes  a  closer  look  at  nearby  contacts  of  interest  as  designated  by  the 
CSG  or  ESG.  The  scenario  begins  with  the  USV  departing  the  HVU  and  traveling  to  the 
nearest  enemy. 

In  PYTHAGORAS,  in  order  for  one  agent  to  be  able  to  investigate  another  agent, 
it  is  necessary  that  they  be  on  opposing  sides.  In  each  of  the  two  ISR  scenarios,  there  are 
two  types  of  opposition,  representing  ships  that  are  merchant  ships  (a  noise  factor),  as 
well  as  the  contacts  from  which  the  friendly  force  is  truly  looking  to  gather  intelligence. 
The  experiments  (discussed  in  detail  in  Chapter  III)  are  set  up  so  that  each  model 
provides  information  on  the  proportion  of  enemies  detected  in  each  scenario  and 
compares  outcomes  between  the  two  ISR  scenarios. 

This  study  also  uses  PYTHAGORAS  to  implement  Eorce  Protection  capabilities 
as  well  as  ISR  capabilities.  Scenario-EP  is  a  scenario  where  the  ratio  of  attackers  to 
defenders  is  high.  The  Eorce  Protection  scenario  has  three  classifications  of  enemies,  for 
PYTHAGORAS  purposes.  As  mentioned  for  the  ISR  scenarios,  there  are  both  neutral 
contacts  and  opposing  forces  within  the  USV  field  of  regard  that  are  not  possible  threats 
to  the  HVU.  The  additional  force  of  enemies  is  threatening,  and  these  hostile  contacts  act 
nearly  simultaneously  to  attack  the  HVU.  Each  USV  opposing  forces  looks  for  their 
nearest  enemy  to  investigate.  The  neutral  contacts  and  non-threatening  enemies  act  as 
noise  factors  for  this  scenario.  The  experiment  is  set  up  so  the  model  will  provide 
information  on  the  proportion  of  enemies  detected,  the  proportion  of  threatening  enemies 
detected,  and  the  number  of  enemies  that  reach  the  HVU. 
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c. 


SCENARIO  DESIGN 


Figure  2.  Screen  Shot  of  Waypoint  Scenario  in  PYTFIAGORAS 

These  models  require  translation  into  the  language  of  PYTHAGORAS  in  order  to 
be  compatible  with  the  real  world  scenario.  The  modeling  platform  uses  pixels  for 
distances,  and  speed  is  in  pixels/time  step.  Properties  such  as  the  speed  of  the  individual 
agents  determine  the  agent’s  position  and  state  after  each  time  step.  Every  scenario  has 
agents  representing  three  forces:  friendly,  neutral,  and  enemy.  Figure  2  displays  the 
agents  in  the  Waypoint  scenario.  Each  force  has  its  unique  properties  that  determine 
movement  and  activity.  The  friendly  force  consists  of  the  HVU  and  all  of  the  USVs. 
Neutrals,  such  as  merchant  vessels,  represent  those  ships  that  are  simply  moving 
randomly  throughout  the  area,  interacting  with  neither  the  enemy  nor  the  friendlies. 
Although  this  force  is  neutral,  PYTHAGORAS  represents  these  contacts  as  enemies,  so 
the  USV  has  reason  to  approach  them. 
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Enemy  agents  are  different  depending  on  the  desired  mission,  ISR  or  FP,  for 
analysis.  Enemy  movements  for  ISR  scenarios  are  as  follows,  in  descending  order  of 
their  desire: 

•  Move  away  from  nearest  enemy  if  closer  than  two  nm, 

•  Maintain  last  course, 

•  Move  randomly  about  the  space. 

The  enemies  in  the  FP  scenario  act  more  hostile  in  order  to  get  to  the  HVEl.  This  is 
explained  in  detail  later.  The  only  movement  desire  the  neutral  agents  possess  is  to  move 
in  a  random  direction.  All  agents  have  the  ability  to  possess  sensors  and  weapons,  and 
specific  speeds,  and  movement  desires.  Movement  desires  can  be  selected  by  four 
different  methods:  highest  desire,  average  desire,  random  desire,  or  the  top  two  desires 
(Bitinas,  2004). 

PYTHAGORAS  requires  each  agent  to  possess  a  weapon  and  a  sensor.  However, 
to  model  agents  without  a  sensor  or  weapon,  the  platform  can  use  dummy  weapons  which 
have  they  have  a  zero-probability  of  kill.  Each  agent  type  in  these  scenarios  possesses  a 
sensor  so  a  dummy  sensor  is  not  necessary.  For  all  scenarios  explored,  the  HVU  inhabits 
the  center.  To  reduce  the  simulation  models’  complexity,  movements  of  each  agent 
(EISV,  enemy  or  neutral)  are  considered  relative  to  the  motion  of  the  HVEl;  therefore,  the 
effect  is  that  of  a  maneuvering  board  or  a  radar  display  onboard  the  HVEl. 

Other  features  of  the  simulation  models  include  the  inputs  into  PYTHAGORAS 
that  remain  constant  throughout  the  simulation  replications  while  varying  factors  and 
scenarios.  Incorporated  features  are  weapon  and  sensor  ranges,  kill  and  detection 
probabilities  in  the  scenarios,  some  agents’  color  state  values,  and  movement  desires. 

Weapons,  firing  and  kill  behaviors  in  PYTHAGORAS  emulate  detection  and 
identification.  “Weapon  range”  defines  the  distance  an  agent  possessing  the  weapon 
must  be  from  the  contact  in  order  to  fire  the  weapon.  This  equates  to  a  detection.  Along 
with  the  range  of  the  weapons,  the  modeler  must  specify  the  probability  of  kill,  Pk,  once 
the  weapon  is  used.  For  all  scenarios  in  this  thesis,  the  range  and  the  Pk  are  constant.  The 
maximum  range  of  the  weapon  is  1  nm.  The  reason  a  USV  has  to  be  so  close  to  “kill”  is 


11 


related  how  kills  equate  to  positive  identification.  Once  a  USV  is  within  1  nm  of  the 
contact,  it  is  “hit,”  and  the  identification  process  is  complete. 

The  Pk  is  set  at  1.0  for  any  range,  calculated  in  PYTHAGORAS  through 
interpolation.  Since  the  time  step  is  72  seconds,  randomness  is  overrun  by  the  time  step. 
For  simplicity,  the  Pk  and  the  maximum  range  of  the  weapon  are  constant.  The 
probability  of  detection,  Pd,  is  similar  to  the  Pk  but  relates  to  the  ability  for  the  sensor  to 
detect  a  contact.  The  optical  sensor  has  a  maximum  range  of  4  nm  and  the  radar  sensor 
has  a  maximum  range  of  16  nm  (Rich,  2004),  that  of  a  commercial  radar  system.  Plots  in 
Figures  3  and  4  show  the  explicitly  stated  probabilities  by  range  for  optical  sensor  and 
radar,  respectively.  The  probabilities  are  an  abstraction  to  show  the  relationship  of  the 
ability  of  each  USV  to  detect  contacts  with  these  sensors.  The  program  linearly 
interpolates  any  range  that  is  not  at  the  stated  distances  to  determine  the  correct 
probability. 


Probability  of  Detection;  Optical 
Sensor 


Distance  from  Agent  (Nautical  Miles) 


Figure  3.  Probability  of  Detection  for  Optical  Sensor 
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PYTHAGORAS  distinguishes  among  friendlies,  enemies  and  neutrals  using  color 
properties.  Each  of  the  three  forces  has  the  same  distinct  color  for  every  scenario. 
Friendlies  are  blue;  the  unidentified  or  alive  enemy  are  solid  red;  “killed”  Enemy  are  red 
circles;  and  neutrals  start  out  as  brown  and  turn  light  blue  after  being  identified.  Color  or 
state  changes  occur  only  when  a  USV  shoots  neutral  agents.  A  neutral  agent’s  color 
changes  so  it  is  no  longer  seen  as  a  potential  enemy  contact,  but  it  is  not  “killed.”  This 
equates  to  a  circumstance  in  which  contacts,  once  identified,  remain  so  throughout  the 
remainder  of  the  scenario.  This  may  be  optimistic.  These  color  properties  are  the  same 
for  each  model  to  facilitate  comparisons. 

Agent  movement,  as  previously  mentioned,  has  values  from  0-100.  The  value 
entered  is  a  number  that  relates  the  particular  desire  to  the  other  desires.  Obviously,  if  the 
movement  desire  is  0,  that  particular  desire  has  no  effect  on  the  agent  and  is  omitted  from 
this  discussion.  This  value  is  only  a  relative  relationship  with  respect  to  the  agent  it  is 
describing.  The  values  are  chosen  only  so  that  the  relationships  among  the  competing 
movement  desires  can  be  shown.  The  relative  importance  of  the  movement  is  the  desired 
relationship  in  the  modeling. 

1,  Scenario-P 

The  area  of  operation  for  this  scenario  represents  a  10  nm  x  10  nm  area  of  ocean 
relative  to  the  HVTJ.  In  PYTHAGORAS,  only  the  EISV  has  weapons  to  “kill”  a  contact. 
The  HVEl,  enemy  and  neutral  agents  do  not  posses  any  weapons.  A  kill  represents 
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correctly  identifying  the  contact,  and  the  likelihood  of  obtaining  a  kill  varies  with  a 
probability  of  kill  (Pk).  Pk  is  0.85  in  Seenario-P  (Quarderer,  2004).  The  feedbaek  from 
prototype  testing  also  indieated  lateney  in  the  data-link  baek  to  the  HVU,  modeled  by 
delaying  the  time  that  the  weapon  can  kill. 

The  sensors  in  the  seenario  are  optical  and  radar.  The  USV  possess  both  the 
optical  and  radar  sensor,  the  HVU  only  has  radar,  and  the  enemy  and  neutral  agents  only 
possess  the  optical  sensor.  The  USV  optical  sensor  represents  the  EO/IR  eamera,  and  has 
perfeet  vision  for  45  degrees  direetly  in  front  and  some  peripheral  vision,  whereas  the 
neutral  and  enemy  sensors  are  merely  visual  cues.  The  probability  of  deteetion  (Pd) 
deereases  as  the  range  to  the  eontact  increases.  The  maximum  range  is  2  nm.  The  radar 
sensor  has  360-degree  eoverage  with  a  maximum  range  of  10  nm  due  to  eommunieations. 

As  observed  for  the  prototype,  the  RF  range  for  the  USV  is  a  5  nm  radius  from  the 
host  platform.  This  limits  the  USV  to  travel  only  in  a  5  nm  area  around  the  HVU.  The 
movement  desires  of  the  USV  are,  in  decreasing  order: 

•  Away  from  leader  (HVU)  if  eloser  than  0.5  nm, 

•  Toward  leader  if  farther  than  5  nm, 

•  Toward  nearest  enemy  if  farther  than  0.2  nm, 

•  Toward  next  waypoint, 

•  Maintain  last  eourse. 

The  prototype  scenario  does  not  have  a  design  matrix  with  replieating  runs.  A 
eonfirmatory  run  is  made  in  order  to  verily  the  seenario  is  representing  the  data  from 
GET. 

2.  Scenario-W 

Scenario-W  implements  a  seenario  where  eaeh  USV  patrols  in  pre-determined 
traeks,  via  waypoints.  Taotieally,  each  should  simply  be  able  to  vigorously  patrol  sectors 
rather  than  follow  specific  paths,  but  the  limitations  of  PYTHAGORAS  did  not  permit 
this  Seetor  Seenario  to  be  developed.  The  waypoints  make  a  “bow-tie”  patrolling  pattern 
near  the  HVU.  The  total  area  represented  is  40  nm  x  40  nm.  As  in  Scenario-P,  the  only 
agent  that  has  a  weapon  is  the  USV,  whieh  “kills,”  or  in  the  terms  of  the  thesis  aecurately 
detects,  the  enemy.  All  three  sides,  friendly,  enemy,  and  neutral,  have  an  optical  sensor 
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that  has  a  45-degree  frontal  view  with  a  view  probability  of  1.0  and  a  0.0  view  probability 
for  any  other  angle. 

The  HVU  agent  possesses  a  radar  sensor  with  a  range  of  16  nm,  equivalent  to  a 
commercial  navigational  radar  system  (Rich,  2004)  and  that  information  is  broadcast  to 
each  USV.  The  broadcast  range  represents  how  far  the  HVU  can  cue  the  USV  with 
information  from  its  radar.  This  property  constrains  each  USV  to  be  within  the  broadcast 
range  in  order  to  receive  the  information  that  the  HVU  is  sending.  This  is  simply  a 
modeling  aspect  that  is  not  intended  to  represent  the  actual  HVU  sends  information  at 
precise  ranges.  The  radar  has  a  360-degree  view  with  a  view  probability  of  1 .0. 

Speed  in  the  simulation  model  is  a  function  of  the  pixel  representation  as  well  as 
the  length  of  the  time  step.  The  speed  among  like  USV  agents  is  set  to  a  tolerance  factor 
of  three.  The  tolerance  factor  is  input  by  the  modeler  and  indicates  the  range  of  the 
property,  such  as  speed,  that  the  agents  in  the  class  can  possess.  For  instance,  if  enemy 
speed  is  10  pixels/time-step,  a  tolerance  factor  of  three  provides  for  the  agents  can  having 
speeds  from  7  to  13  knots.  This  number  allows  speed  to  vary  slightly  during 
experimentation.  Since  the  range  of  speed  is  from  1-20  units,  a  factor  of  three  can  be 
used  consistently  over  the  varied  speeds.  Each  USV  starts  at  the  HVU,  the  center  of  the 
modeling  area,  and  travels  toward  the  waypoints.  Since  speeds  are  not  constant,  the 
scenario  represents  average  speeds. 

The  constant  radius  modeled  for  each  USV  is  20  nm.  The  range  is  just  over  the 
visual  horizon  and  past  the  16  nm  radar  range.  Experimenting  with  this  range  shows 
whether  it  is  desirable  to  stretch  the  limits  beyond  the  line  of  sight.  This  is  one  of  the 
factors  that  varies  throughout  the  runs  of  the  simulation.  The  movement  desires  of  each 
USV  are,  in  decreasing  order  of  desire: 

•  Toward  the  next  waypoint  within  a  distance  of  1 .6  nm, 

•  Toward  the  nearest  enemy  if  farther  than  0.4  nm, 

•  Away  from  the  closest  unit  member  if  closer  than  1  nm, 

•  Away  from  the  HVU  if  closer  than  1  nm,  and 

•  Toward  the  HVU  if  farther  than  1  nm. 
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The  first  two  desires  are  given  equal  weighting  and  the  movement  method  is  used  “top 
two  desires”,  which  means  that  the  agent  only  looks  at  the  two  highest  desire  values  to 
execute  the  next  movement.  If  conditions  of  the  highest  values  are  not  met,  the  agent 
looks  at  the  next  highest  desire  value,  allowing  the  USV  to  seek  for  enemies  as  well  as 
proceed  to  the  next  waypoint.  The  third  desire  prevents  the  US  Vs  from  clustering 
together  and  going  after  the  same  contacts.  Forcing  the  USVs  apart  makes  them  operate 
separately.  The  fourth  desire  is  to  “jump  start”  the  simulation.  All  of  the  USV  would 
otherwise  start  at  the  center  and  stay  put  without  this  movement  desire.  The  last  is  a  very 
low  desire  level  and  if  all  of  the  other  conditions  cannot  be  met,  the  USVs  go  back  to  the 
HVU. 

3.  Scenario-I 

The  Interceptor  scenario  is  a  representation  of  USV  operating  in  a  random 
manner,  whereby  each  conducts  cooperative  searches  for  contacts  within  their  optical 
scope  or  that  of  the  HVU  radar.  The  search  area  contains  neutral  contacts  and  enemy 
contacts,  as  well  as  the  HVU.  The  USVs  deploy  from  the  HVU  at  the  start  of  the 
simulation  and  immediately  start  searching  for  the  nearest  enemy.  When  the  USV  is 
within  the  stated  range  of  identifying  the  contact,  the  contact  is  removed  from  the 
possible  contacts  that  can  be  explored.  In  order  for  the  USV  agents  in  PYTHAGORAS 
to  desire  investigating  non-threatening  contacts,  the  neutral  contact  is  designed  as  an 
enemy.  As  in  tactical  operations,  the  USV(s)  would  attempt  to  identify  not  only  the 
enemies,  but  any  unknown  contact  that  is  within  the  chosen  zone  of  the  HVU. 

The  USV  agents  in  the  scenario  possess  optical  and  radar  sensors.  The  optical 
sensor  has  a  perfect  view  45 -degree  in  front  of  the  agent  with  no  peripheral  vision.  The 
HVU  possesses  the  radar  sensor,  which  has  a  360-degree  view  and  a  range  of  16  nm. 
The  neutral  and  enemy  agents  only  possess  the  optical  sensor.  In  descending  order,  the 
movement  desires  of  each  USV  in  the  Interceptor  scenario  are; 

•  Toward  the  nearest  enemy  as  seen  in  the  scenario  (this  sends  the  USV  after 
the  contacts), 

•  Away  from  the  closest  unit  member  (this  prevents  the  USV  from  staying  in  a 
cluster). 
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•  Away  from  the  leader  (HVU)  if  closer  than  one  nm  (this  jump-starts  the 
search),  and 

•  Toward  the  leader  (HVU)  if  farther  than  one  nm  (if  no  other  movement 
desires  exist,  this  sends  the  USV  back  to  the  HVU). 

Once  again  the  movement  method  uses  the  “top  two  desires”  of  the  values  input  into  the 
simulation  model.  This  is  an  ISR  scenario  and  it  follows  the  descriptions  and 
assumptions  of  Scenario-W  for  the  movement  desires  for  the  neutral  and  enemy  contacts 
in  the  scenario.  The  verification  of  Scenario-I  is  Random  Search  Theory  applied  in 
Chapter  IV. 

4.  Scenario-FP 

The  Force  Protection  scenario  is  designed  to  determine  the  benefits  of  the  USV 
when  there  is  an  imminent  threat  to  the  HVU.  As  in  the  previous  scenarios,  each  USV 
searches  for  the  nearest  contact  in  order  to  identify  whether  or  not  it  is  a  threat.  The 
weapons  and  sensors  that  the  USV  and  the  other  contacts  possess  are  the  same  as  in  the 
ISR  scenarios.  The  difference  in  this  scenario  from  the  ISR  scenarios  is  how  the  enemies 
are  defined.  The  purpose  is  to  see  the  effect  of  each  USV  when  some  of  the  enemy 
contacts  are  directly  targeting  the  HVU.  The  neutral  and  enemy  contacts  in  Scenario-FP 
are  still  implemented  as  in  the  ISR  scenarios;  however  there  is  an  additional  type  of 
enemy  in  the  FP  situation.  The  additional  enemy  is  threatening  and  goes  toward  the 
HVU  to  attack.  The  actual  attack  on  the  HVU  is  not  modeled,  as  the  focus  of  this  study 
is  on  USV  activity.  Modeling  the  attack  and  how  the  USV(s)  respond  could  be  a  topic 
for  future  research. 

The  movement  desires  of  the  threatening  enemy  are  to  make  it  to  the  HVU 
without  being  detected  by  the  USVs.  They  are  hostile  toward  the  HVU  and  the  USVs, 
and  do  not  cluster  tightly  but  to  join  together  outside  a  range  of  four  nm.  The 
PYTHAGORAS  desires  are  listed  in  descending  order  of  desire: 

•  Toward  next  waypoint  (HVU), 

•  Toward  nearest  enemy  (USV(s)), 

•  Away  from  nearest  enemy  if  farther  than  two  nm. 
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•  Away  from  Closest  unit  Member  if  closer  than  one  nm, 

•  Toward  closest  unit  member  if  farther  than  four  nm, 

•  Maintain  last  course, 

•  Select  Random  Direction. 


To  affect  tactically  challenging  threat  profiles,  the  simulation  model  makes  use  of  a 
“highest  desire”  movement  method  among  these  options.  The  first  two  each  have  the 
highest  desire,  the  third,  slightly  less,  and  the  fourth,  fifth,  and  sixth  are  all  equal.  The 
last  two  are  carried  over  from  the  enemy  characteristics  in  the  ISR  scenarios  and  the  non¬ 
threatening  enemies  in  the  FP  scenario.  USV  movement  desire  is  exactly  the  same  as  for 
the  Interceptor  scenario. 


D,  METHODOLOGY 

No  experiment  is  complete  unless  there  is  output  of  value  for  analysis.  The  value 
is  a  function  of  the  factors  and  the  factor  levels  explored  in  the  experiment  (Chapter  III) 
as  well  as  the  measures  of  effectiveness  (MOEs)  that  are  collected.  The  following  MOEs 
appear  to  be  most  relevant,  even  though  PYTHAGORAS  is  able  to  implement  a  variety 
of  MOEs.  The  ISR  scenarios  explore  the  proportion  of  enemies  detected.  The  EP 
scenario  also  examines  the  proportion  of  enemies  detected.  Since  there  are  two  types  of 
enemies,  there  are  two  distinct  MOEs  for  each  type  of  enemy — the  overall  proportion  of 
enemies  detected  and  the  proportion  of  threatening  enemies  that  are  detected.  The  final 
MOE  to  be  evaluated  is  the  number  of  threatening  enemies  that  reach  the  HVU.  Each 
MOE  is  evaluated  individually  against  the  factors  for  the  respective  scenario,  and 
analyzed  using  regression  methods.  The  results  yield  significant  factors  and  interactions 
for  each  particular  MOE.  Eurther  validation  of  the  important  factors  confirms  the  results. 

1.  MOEs  Implemented 

a.  Proportion  of  Enemy  Detections 

The  numbers  of  each  type  of  agent  are  known,  since  they  are  inputs  to 
PYTHAGORAS  at  the  beginning  of  each  run.  The  output  from  PYTHAGORAS  returns 
the  number  of  detected  enemies  at  the  end  of  each  run,  yielding  a  proportion  of  enemies 

detected.  The  proportion  of  detections  is  essential  to  the  ISR  scenarios  as  well  as  the  EP 
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scenarios.  For  example,  it  is  tactieally  important  to  know  eonditions  in  whieh  sending 
out  a  USV  would  enable  deteetion  of  only  a  small  proportion  of  enemies.  It  is  a  waste  of 
an  asset  if  the  proportion  of  detections  is  so  low  as  not  to  enhance  the  mission.  It  is 
taotieally  important  to  specify  the  condition  under  whieh  the  HVU  deploys  the  USV  to 
interdict  contacts.  This  study  helps  define  those  eonditions.  It  is  anticipated  that  as  the 
number  of  USVs  inerease,  the  proportion  of  detections  will  initially  increases  but 
eventually  levels  off. 

Foree  Protection  is  most  critical  at  close  ranges  to  the  HVU.  Knowledge 
of  the  probability  of  detection,  as  well  as  the  current  number  of  deteetions,  eould  be  used 
to  develop  an  estimate  of  the  proportion  of  unidentified  enemies  in  the  area.  This  type  of 
information  is  of  great  value  to  tactical  decision  makers. 

b.  Proportion  of  Detections  against  Threatening  Enemies 

This  MOE  is  calculated  similarly  to  MOE  (a),  the  proportion  of  deteetions 
of  enemies  that  are  threatening  out  of  the  total  number  of  enemies.  The  real  world  actual 
number  of  enemies  is  unknown,  therefore  we  don’t  know  the  probability  of  detection  give 
the  number  of  enemies.  The  benefits  of  this  study  are  that  we  know  the  values  of  the  all 
the  factors  so  we  ean  generate  estimated  probabilities  of  detection  given  the  number  of 
enemies  for  specific  conditions. 

c.  Number  of  Threatening  Enemies  that  Reach  the  HVU 

Since  the  threatening  enemies  are  designed  to  move  toward  the  HVU,  this 
study  must  show  how  many  of  these  threatening  enemies  meet  their  goal.  This  MOE 
should  give  the  decision-makers  the  information  necessary  to  decide  what  USV  options 
to  exercise  when  planning  a  EP  mission.  This  MOE  is  important  in  determining  an 
effective  numbers  and  employment  of  USVs  for  Eorce  Proteetion  missions. 
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III.  DESIGN  OF  EXPERIMENTS 


A.  LATIN  HYPERCUBE  DESIGN 

Correct  design  of  experiments  enables  efficient  analysis  of  many  factors  at 
multiple  levels.  A  design  point  represents  the  combination  of  one  particular  level  for 
each  factor  in  the  simulation  model.  In  general,  choosing  the  values  for  the  design  points 
can  be  challenging,  particularly  when  the  number  of  factors  is  large,  such  as  in  this  study. 
For  example,  observing  11  factors  at  three  discrete  levels  would  require  or  177,147 
design  points,  to  be  run  at  least  ten  times  each  for  randomness,  for  a  total  of 
approximately  1.8  million  runs.  Observing  1 1  factors  at  ten  discrete  levels  would  require 
lO”  or  100  billion  design  points  to  be  run  at  least  10  times  each  for  randomness,  for  a 
total  of  one  trillion  runs.  In  contrast,  the  designs  used  to  explore  the  three  scenarios  in 
this  thesis  require  less  than  12,000  runs  combined,  yet  allow  for  up  to  33  discrete  levels 
for  each  factor. 

This  base  design  for  the  study  is  a  Nearly  Orthogonal  Latin  Hypercubes  (NOLH), 
which  takes  at  most  eleven  factors  and  varies  each  factor  across  up  to  33  different  levels. 
Using  a  Microsoft  Excel  spreadsheet  developed  by  Professor  Susan  Sanchez  (Sanchez, 
2004)  based  on  Cioppa’s  (2002)  designs,  it  takes  the  minimum  and  maximum  value  of 
each  factor  as  inputs,  and  specifies  combinations  of  integer  values  within  the  factor 
ranges  as  outputs.  The  resultant  matrix  is  a  set  of  design  points  that  are  generated  so  that 
the  columns  are  virtually  orthogonal  to  each  other,  defined  as  a  correlation  less  than 
|0.03|,  simplifying  the  process  of  adding  and  removing  terms  to  a  regression  model. 
When  all  factors  are  independent,  an  individual  factor  has  the  same  effect  on  the  response 
variable  whether  or  not  other  factors  are  included  in  the  regression  model. 

The  NOLH  design  is  devised  so  the  design  space  is  covered  fully.  The  factor 
levels  are  varied  to  maximize  the  coverage.  Good  space-filling  permits  complex  analyses 
to  be  computed  (Sanchez  and  Lucas,  2002).  This  study  appends  together  four  NOLH 
matrices,  in  order  to  get  better  space-filling  and  sufficiently  reduced  correlation.  The 
middle  run  for  each  of  the  four  matrices  is  the  exact  same  design  point  (a  function  of  the 
calculations  to  generate  the  values  in  the  columns),  so  there  are  a  total  of  129  distinct 
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design  points  run  30  times  eaeh  for  a  total  of  3,870  runs  for  each  scenario.  This  provides 
structure  so  that  the  data  can  be  compressed  for  each  design  point  and  for  analysis  as  129 
independent  responses. 

The  runs  were  conducted  on  a  cluster  of  machines  at  the  MITRE  organization  in 
Woodbridge,  VA.  Cleaning  the  data  for  analysis  exposed  errors  in  the  compilation  of  the 
data  files.  Omitting  these  four  associated  design  points  yielded  a  correlation  matrix 
producing  a  maximum  correlation  of  |0.06|.  Even  though  this  is  not  below  |0.03|,  it  is  still 
sufficiently  low  for  us  to  consider  the  factors  nearly  orthogonal.  These  replications  help 
determine  which  of  the  varied  factors  are  important  for  specific  MOEs.  Since  the 
composition  of  the  ISR  and  FP  scenarios  are  slightly  different,  the  design  matrices  are  not 
exactly  the  same.  However,  each  look  at  parallel  factors.  The  design  matrices  are 
explained  in  detail  in  the  following  two  sections. 

1,  Explanation  of  Variable  Factors  for  ISR  Scenarios 

Ten  factors  are  varied  in  PYTHAGORAS  units  for  the  ISR  scenario.  The  factors 
are  shown  in  Table  1,  along  with  their  minimum  and  maximum  tactical  values  for  the 
simulation  experiments.  In  the  remainder  of  this  section,  each  factor  is  also  explained  in 
its  PYTHAGORAS  terms. 


Table  1.  ISR  Scenarios  NOLH  Design,  10  Factors,  33  Design  Points 


Factor 

Minimum  Value 

Maximum  Value 

USV  Speed 

2  knots 

40  knots 

Neutral  Speed 

2  knots 

40  knots 

Enemy  Speed 

2  knots 

40  knots 

Sea  State 

1 

3 

Number  of  USV(s) 

1 

24 

Number  of  Contacts 

1 

250 

%  Enemies 

10% 

25% 

USV(s)  Permitted  Range  Ifom  HVU 

1  nm 

20  nm 

Camera  Range 

1  nm 

10  nm 

Length  of  Simulation 

30 

500 

The  speed  of  each  type  of  agent  ranges  from  1-20  pixels  per  time  step,  which 
converts  to  2-40  knots.  Each  time  step  is  72  seconds  long.  The  maximum  speed  of  each 
USV  is  based  on  that  of  the  RHIB,  the  base  structure  for  the  USV.  The  maximum  speed 
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of  a  RHIB  is  40  knots  (Navy  Fact  File,  2004).  The  smallest  time  step  feasible  to  permit 
the  eonversion  to  40  knots  in  PYTHAGORAS  is  72  seeonds.  Two  knots  is  the  minimum 
at  which  each  time  step  is  not  so  long  that  resolution  of  the  seenario  is  lost.  If  the  range 
were  one  knot,  the  time  step  would  need  to  be  twiee  as  long,  signifieantly  deereasing  the 
fidelity  of  the  seenario  sinee  detections  can  take  plaee  in  less  than  two  minutes.  Both  the 
neutral  and  enemy  eontacts  are  given  the  same  range  for  the  respeetive  speeds. 

The  next  factor  varied  is  the  sea  state,  accomplished  in  PYTHAGORAS  by  the 
ehanging  the  movement  faetor  of  the  terrain.  The  movement  faetor  is  a  number  between 
zero  and  one.  In  order  to  implement  the  NOTH  design,  this  faetor  is  sealed  by  10  to 
eonform  to  spreadsheet  design  points  given  in  integer  value.  The  range  of  the  faetor  in 
PYTHAGORAS  is  0.5 -1.0  (5-10  in  the  design  matrix).  The  aetual  range  of  sea  states  is 
states  1-3.  Above  sea  state  three,  a  RHIB  is  highly  unstable,  its  sensors  are  rendered  less 
effective,  and  the  hostile  and  neutral  agents  suffer  similarly  in  their  ability  to  maneuver. 
Looking  at  operations  in  the  heightened  sea  states  is  an  opportunity  for  further  researeh. 
An  overview  of  sea  state  classifieation  and  translation  to  PYTHAGORAS  appears  in 
Table  2. 

Table  2,  Sea  State  Definition  for  Pythagoras  (from  Definition  of  Sea  State) 


Pythagoras 

Movement 

Factor 

Sea  State 

Wave 

Height 

(Ft) 

0.9-1.0 

1 

0.5-1 

0.7-0.8 

2 

1.5-2.5 

0.5-0.6 

3 

3-4 

In  the  scenario,  an  HVU  is  assigned  a  Squadron  of  USVs,  which  consists  of  six 
detachments  of  four  USVs  each,  totaling  24  USVs  per  HVU  (Rieei,  2002).  At  one 
extreme,  it  is  desired  to  know  the  outcome  when  USVs  are  outnumbered  by  eontaets,  and 
in  faet  more  enemy  eontaets  than  USVs.  Otherwise,  the  HVU  would  be  most  likely  to  be 
able  to  handle  the  few  eontaets.  To  explore  this  broad  range  of  alternatives,  the  number 
of  eontaets  varies  from  1  to  250,  where  the  enemy  agents  make  up  10-25%  of  all 
potential  eontacts.  These  two  factors  determine  how  many  neutral  and  enemy  agents  are 
in  the  seenario. 
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The  optical  sensor  has  a  maximum  range  which  defines  how  far  away  the  agent 
possessing  the  sensor  can  detect  a  contact.  By  changing  the  maximum  range  of  the 
optical  sensor,  the  time  available  for  the  PYTHAGORAS  agent  to  “kill”  the  contact 
varies  as  well;  it  varies  over  the  interval  1-10  nm,  or  25-250  pixels.  As  the  sensor  range 
increases,  the  number  of  detections  should  increase  as  well. 

The  permitted  range  of  the  US  Vs  from  the  HVU  varies  to  assess  how  close  the 
USV  should  be  stationed  with  respect  to  the  HVU.  The  range  of  the  camera  sensor  varies 
throughout  the  simulation.  This  range  determines  how  close  the  USV  must  be  to  the 
contact  in  order  to  get  an  accurate  identification  of  the  contact  at  hand.  The  sensor  range 
varies  from  Inm  to  20nm. 

The  final  design  factor  under  consideration  is  the  time  on  station,  which 
determines  availability  for  each  USV  to  complete  the  mission.  The  projected  time  on 
station  for  the  USV  prototype  is  six  hours.  Utilizing  the  72  second  time  step,  the  length 
of  the  simulation  runs  for  36  minutes  up  to  10  hours.  If  the  analysis  reveals  a  high 
correlation  between  the  MOEs  and  the  endurance,  it  would  be  beneficial  to  be  able  to 
lengthen  the  time  a  USV  can  remain  on  station. 

2,  Explanation  of  Variable  Factors  for  FP  Scenario 

All  of  the  factors  varied  in  the  ISR  scenarios  are  also  implemented  in  the  FP 
scenario  with  the  exception  of  time  on  station.  Scenario-FP  has  an  extra  factor, 
threatening  enemies,  that  the  ISR  scenarios  do  not.  This  necessitates  an  extra  speed 
factor  for  the  design  as  well  as  and  additional  MOE  (percentage  of  threatening  enemies 
out  of  the  total  number  of  enemies)  for  the  analysis.  For  the  purpose  of  preserving  the 
NOFH  matrix,  the  time  on  station  factor  is  removed  and  kept  constant  at  10  hours  for 
every  run,  and  the  threatening  enemy  speed  and  percentage  of  enemies  that  are 
threatening  factors  are  added  to  obtain  the  FP  design  matrix.  The  speed  factor  remains 
between  2-40  knots  as  it  is  for  all  other  agents  in  the  ISR  scenarios,  and  the  same 
justification  applies.  The  overall  number  of  enemies,  threatening  and  non-threatening,  is 
10-25%  of  the  total  number  of  contacts,  which  varies  from  1  to  500.  The  threat 
constitutes  between  10-90%  of  the  total  number  of  enemies  so  that  involving  both  few 
and  many  threatening  contacts  are  examined.  The  factors  of  the  optical  sensor  range  and 

the  distance  permitted  from  the  HVU  are  the  same  as  for  the  ISR  scenarios.  These  eleven 
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factors  and  the  real  world  ranges  of  each  factor  for  the  Foree  Protection  scenario  are 
shown  in  Table  3. 


Table  3.  FP  Scenario  NOLH  Design,  11  Factors,  33  Design  Points 


Factor 

Minimum  Value 

Maximum  Value 

USV(s)  speed 

2  knots 

40  knots 

Neutral  Speed 

2  knots 

40  knots 

Threatening  Enemy  Speed 

2  knots 

40  knots 

Non- threatening  Enemy  Speed 

2  knots 

40  knots 

Sea  State 

1 

3 

Number  of  USV(s) 

1 

24 

Number  of  Contacts 

1 

500 

%  Enemies 

10% 

25% 

USV(s)  Permitted  Range  from  HVU 

1  nm 

20  nm 

Camera  Range 

1  nm 

10  nm 

%  Hostile  Enemies 

0% 

100% 

B,  TACTICAL  INTERPRETATION 

As  mentioned  in  the  previous  seetion,  the  research  and  experimentation  is  done  so 
that  it  can  be  applied  to  the  needs  of  the  Fleet.  To  aecomplish  this,  factors  that  can  be 
eontrolled  by  the  HVU,  and  the  noise  factors  that  cannot,  are  varied  in  the  simulation 
experiments.  Currently,  the  prototype  for  the  USV  has  limitations,  such  as  the  range  it 
can  travel  from  the  HVU  and  the  sensor  ranges.  If  the  analysis  in  the  next  seetion  proves 
a  need  for  the  USV  to  hold  properties  that  are  beyond  the  current  capabilities,  these 
capabilities  should  be  implemented  to  achieve  optimal  functionality. 

There  are  also  factors  such  as  the  number  of  contacts  and  their  speeds,  and  the  sea 
state  in  the  1600  sq-nm  area  of  ocean  looked  at  in  the  scope  of  this  thesis  that  are 
uncontrollable  by  the  HVU.  The  eontrollable  factors  are  the  number  of  USVs  and  the 
speed  of  the  USVs.  The  ultimate  desire  is  to  be  able  to  observe  an  approximate  number 
of  contacts  in  the  tactieal  scenario  at  a  given  sea  state  and  know  how  many  USVs  should 
be  employed  to  investigate.  Ideally,  the  results  in  this  thesis  will  provide  useful 
information  to  the  Fleet  for  tactical  operations. 
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IV.  EXPERIMENTATION  RESULTS,  COMPARISONS,  AND 

INSIGHTS 


A,  ANALYSIS  APPROACH 

There  are  several  tools  available  to  analyze  simulation  output  data.  Multiple 
linear  regression  is  the  first  analytical  tool.  If  the  regression  is  not  sufficient,  other  means 
may  be  employed  for  classifying  the  factors  in  terms  of  the  MOE.  Using  regression,  we 
look  at  the  factors  varied  in  the  Design  of  Experiments  (DOE),  the  predictor  (regressor) 
variables,  and  each  of  the  MOE’s  as  a  response  to  the  predictors.  The  process  seeks  to 
identify  factors,  quadratic  effects,  and  interactions  that  are  significant  in  explaining 
variation  in  the  MOEs. 

Eor  this  study,  the  overall  regression  model  is  considered  statistically  significant  if 
the  p-value  is  less  than  0.05.  Individual  terms  in  the  model  (main  effects  for  a  factor, 
quadratic  effects,  or  interaction  effects)  are  determined  to  be  statistically  significant  if  the 
p-value  is  less  than  0.05.  The  smaller  the  p-value  is,  the  stronger  is  the  evidence  that  a 
relationship  exists  between  the  term(s)  and  the  MOE . 

One  metric  for  determining  whether  a  regression  model  is  a  “good”  model  is  the 
R-squared  value.  R-squared  is  the  proportion  of  the  variance  in  the  response  that  is 
explained  by  the  terms  included  in  the  model.  A  full  model  includes  all  of  the  factors,  the 
factors  squared,  and  the  two-way  factor  interactions  as  explanatory  terms.  This  involves 
65  terms  for  the  ISR  scenarios  and  77  terms  for  Scenario-EP.  When  trying  to  determine 
what  terms  to  include,  a  high  R-squared  is  desirable,  indicating  the  response  can  be 
closely  predicted;  having  a  minimal  number  of  predictors  is  also  desirable  so  that  the 
actual  variance  of  the  response  is  at  a  minimum  (Montgomery,  2001).  Also,  the  cost  of 
controlling  more  factors  is  usually  directly  related  to  the  number  of  factors — another 
reason  to  keep  the  model  simple. 

There  are  several  ways  to  look  at  a  full  model  and  determine  which  factors  should 
be  eliminated.  This  can  be  done  by  hand  or  by  utilizing  software  to  do  the  process.  This 
analysis  uses  IMP™  software,  which  has  a  stepwise  regression  procedure.  The  forward 
regression  option  starts  with  no  predictors  in  the  model  and  adds  a  term  if  it  has  a  small 
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p-value.  Backwards  elimination  starts  with  all  predictors  and  sequentially  removes  them 
if  the  p-value  is  large.  The  mixed  (or  stepwise)  option  does  both  backwards  elimination 
and  forward  selection  simultaneously.  Usually,  entering  and  exiting  criteria  are  set  to 
avoid  cycling  so  it  is  harder  to  remove  a  predictor  after  it  has  been  added  to  the  model 
(Montgomery,  2001).  We  use  the  mixed  option  of  the  stepwise  regression  offered  by  the 
software  to  do  the  analyses.  This  yields  models  with  fewer  terms  than  the  associated  full 
models  while  still  achieving  high  R-squared  values. 

The  model  suggested  by  the  stepwise  regression  procedure  can  be  modified 
manually,  using  inference  tests  of  the  significance  of  regression  and  the  significance  of 
the  individual  coefficients.  Further  details  are  available  in  any  statistical  text  (e.g., 
Montgomery,  2001),  but  we  summarize  the  tests  here.  The  significance  of  regression  test 
has  the  following  null  and  alternative  hypotheses: 

//o:A=A=-  =  A=0 

p  for  at  least  one  j 

where  k  is  the  last  term  in  the  model  and  j  is  any  term  from  1  to  k.  A  rejection  of 
indicates  that  there  is  at  least  one  predictor  that  has  a  significant  contribution.  The 
method  for  determining  a  rejection  is  if  the  p-value  is  less  than  a  significance  level  a, 
usually  a=0.05.  This  p-value  is  found  in  the  AN OVA  table  summarizing  the  model. 

The  other  type  of  test  for  significance  is  on  the  individual  coefficients  in  the 
regression.  The  null  and  alternative  hypotheses  for  this  test  are  below: 

H,:f3.=0 

where  Pj  is  the  coefficient  for  a  specific  potential  predictor.  A  rejection  of  the  null 

hypothesis  for  this  test  means  the  associated  term  should  remain  in  the  model;  if  there  is 
sufficient  evidence  to  reject  the  null  hypothesis,  then  the  term  is  be  removed.  The 
method  for  determining  whether  or  not  to  reject  the  null  hypotheses  also  involves  a  p- 
value,  reflected  in  the  Prob<|t|  column  in  the  table  of  coefficients  for  the  model.  Again,  a 
typical  criterion  is  to  reject  the  null  hypothesis  if  the  p-value  is  less  than  0.05. 
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If  linear  regression  is  not  significant  or  does  not  provide  a  good  fit,  partitioning  or 
regression  trees  can  help  determine  important  factors.  This  process  explores  which 
factors  can  be  split,  and  at  which  factor  level  the  split  should  occur.  A  candidate  factor  is 
one  that  has  the  largest  sum  of  squares  value.  The  output  of  the  regression  trees  includes 
the  mean  and  standard  deviation  of  the  response  variable,  given  the  conditions  at  the 
levels  where  the  factors  split. 

The  goals  of  the  both  types  of  analyses  include  determining  relatively  simple 
models  that  relate  the  MOEs  to  their  significant  factors.  This  enables  the  analyst  to  make 
predictions  of  the  MOE  at  particular  combinations  of  factor  values,  and  to  relate  the 
factors  to  the  MOE  for  decision  makers. 

B,  ANALYSES 

1.  Scenario-W  Analysis 

Scenario-W  analysis  begins  by  constructing  a  formulation  that  considered  only 
the  terms  associated  with  the  factors  controllable  by  the  HVEl.  Stepwise  regression 
assists  in  determining  which  of  these  controllable  factors  are  significant.  This  function  is 
in  JMP™,  evaluating  all  potential  terms  at  the  significance  levels  for  exiting  and  entering 
the  model  at  0.10  and  0.05,  respectively.  Initially,  we  consider  a  model  that  contains  only 
terms  involving  the  factors  that  are  controllable  by  the  HVU  to  see  if  situational 
awareness  is  required  to  accurately  predict  the  MOE.  These  terms  are:  speed  of  the 
USV,  number  of  USVs,  permissive  range  from  the  HVU  (also  known  as  the  combat 
radius),  camera  range,  and  the  simulation  run-time,  representing  the  available  time  on 
station.  Two-way  interactions  and  quadratic  effects  are  also  included.  Stepwise 
regression  on  these  terms  returns  a  model  that  explains  64.06%  of  the  variance.  The 
main  effects  included  are  the  number  and  speed  of  the  USV(s),  the  camera  range  and  the 
simulation  time.  No  interaction  terms  are  included,  but  the  quadratic  effects  of  the 
number  of  USVs  and  the  camera  range  are  significant.  The  regression  is  significant  since 
the  p-value  is  less  than  0.001  for  the  null  hypothesis.  As  depicted  in  Eigure  5,  the  actual 
versus  predicted  values  do  not  have  a  tight  cohesion,  or  a  strong  fit,  for  these  data  points. 
An  indicator  for  seeking  a  better  model  is  the  residual  plot,  Eigure  6,  which  shows  that 

the  variance  is  not  constant.  The  variance  seems  to  increase  up  to  predictions  of  0.4,  then 
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become  stable.  However,  there  is  a  wide  gap  for  the  residuals  at  predictions  between  0.7 
and  0.8.  With  less  than  two-thirds  of  the  MOE  variation  explained  by  this  model,  a  better 
explanation  may  be  achievable.  While  the  initial  model  may  be  of  some  use  to  decision 
makers,  adding  terms  involving  uncontrollable  factors  may  improve  the  fit  and  residual 
plots  and  provide  additional  insight. 

0.9 
0.8 
0.7 
0.6 
0.5 
0.4 
0.3 
0.2 
0.1 
0 

.0  .1  .2  .3  .4  .5  .6  .7  .8  .9 

Figure  5.  Actual  vs.  Predicted  Responses  for  Significant  Controllable 

Factors  Model  (Scenario-W) 
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Figure  6.  Residual  vs.  Predicted  Responses  for  Significant  Controllable  Factors 

Model  (Scenario-W) 

A  new  model  is  generated  after  including  all  65  possible  terms;  the  main  effects, 
two-way  interactions  and  quadratic  effects.  Figure  7  shows  the  actual  simulation  values 
versus  the  predicted  values  for  this  “full”  model.  The  data  points  are  clustered  much 
more  tightly,  which  shows  a  good  fit  of  the  data  to  the  model.  The  regression  is 
significant  and  the  explanation  of  the  variance  has  increased  to  93.71%  from  64.06%. 
The  residual  plot,  shown  in  Figure  8,  also  shows  that  there  are  no  model  defects. 
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Figure  7.  Actual  vs.  Predicted  Responses  for  Full  Model  (Scenario-W) 
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Figure  8.  Residual  vs.  Predicted  Responses  for  Full  Model  (Scenario-W) 

Even  though  having  93.71%  of  the  variance  explained  by  these  65  terms  is  great, 
a  decision-maker  will  find  it  much  easier  to  examine  a  model  with  fewer  terms.  Using 

the  stepwise  function  as  before,  the  stepped  model  has  only  a  slightly  lower  R-squared 

value  (92.22%)  but  the  number  of  terms  drops  to  22 — a  considerable  improvement  from 
the  initial  65.  Figure  9  shows  the  actual  versus  predicted  values.  Compared  to  Figure  7, 
the  stepped  model  is  still  a  good  fit.  Looking  at  the  residual  plot  (Figure  10),  there  still 
does  not  seem  to  be  a  severe  model  defect  since  the  residual  plot  is  approximately  a 
horizontal  band  and  there  are  no  outliers  in  the  data. 
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Figure  9.  Actual  vs.  Predicted  Responses  for  Stepped  Model  (Scenario-W) 
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Figure  10.  Residual  vs.  Predicted  Responses  for  Stepped  Model  (Scenario-W) 

The  criteria  we  used  for  the  stepwise  procedure  made  it  more  difficult  for  terms  to 
leave  the  model  once  they  have  entered,  so  it  may  be  possible  to  remove  some  terms 
manually  without  greatly  reducing  the  amount  of  explained  variance.  We  removed  terms 
where  the  test  on  individual  regression  coefficients  is  not  rejected  (p-value  greater  than 
0.05).  This  third  (and  final)  model  uses  only  11  terms,  yet  explains  88.99%  of  the 
variance  of  the  proportion  of  enemies  that  are  accurately  identified.  The  regression  is 
still  significant.  Figure  1 1  shows  the  actual  vs.  predicted  plot  for  this  final  model  for  the 
Waypoint  scenario.  The  residual  plot  in  Figure  12  shows  that  the  model  still  has  no 
major  defects.  A  few  points  (circled  in  Figure  12),  were  looked  at  as  possible  outliers, 
but  all  were  within  three  standard  deviations  of  the  predicted  value. 
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Figure  1 1 .  Actual  vs.  Predicted  Responses  for  Final  Model  (Scenario-W) 


Figure  12.  Residual  vs.  Predicted  Responses  for  Final  Model  (Scenario-W) 

The  terms  in  the  final  model  are  shown  in  the  Table  4  below,  along  with  the 
coefficients,  their  standard  errors,  and  their  p-values.  All  of  the  significant  controllable 
factors  from  the  initial  model  are  in  the  final  model. 


Table  4,  Coefficients  in  the  Final  Model  (Scenario-W) 


Term 

Coefficient 

Std  Error 

P- value 

Intercept 

-0.108938 

0.03293658 

0.0013 

Speed:  USV(s)  (knots) 

0.00214313 

0.0005882 

0.0004 

Speed:  Enemy  (knots) 

0.00853616 

0.00059336 

0.0000 

Number  USV(s) 

0.00988934 

0.00096962 

0.0000 

Camera  Range  (nm) 

0.0255352 

0.00247689 

0.0000 

Simulation  Length  (minutes) 

0.00070397 

0.00003916 

0.0000 

(Speed:  USV(s)  -21. 104)* (Number  USV(s)-12.528) 

0.00022515 

0.0000923 

0.0163 

(Speed:  Enemy  -20.832)*(Simulation  Length  -317.338) 

0.00001972 

0.00000361 

0.0000 

(Speed:  USV(s)  -21.104)*(Speed:  USV(s)  -21.104) 

-0.0001938 

0.00006154 

0.0021 

(Number  USV(s)-12.528)*(Number  USV(s)-12.528) 

-0.0009239 

0.00017693 

0.0000 

(Camera  Range  -5.46848)*(Camera  Range  -5.46848) 

-0.0068333 

0.00108399 

0.0000 

(Simulation  Length  -317.338)*(Simulation  Length  -317.338) 

-0.0000014 

2.96E-07 

0.0000 
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Table  4  shows  that  there  are  two  significant  interactions  in  the  model.  Taking  a 
look  at  each  interaction  through  either  the  contour  plots  or  the  prediction  profiler  reveals 
how  the  combination  of  the  two  factors  affects  the  proportion  of  the  enemies  detected  in 
the  simulation.  Figure  13  shows  a  matrix  of  the  interactions  included  in  the  final  model. 
Each  sub-graph  shows  the  impact  on  the  predicted  MOE  of  changing  the  row  factor  from 
its  lowest  to  highest  level  for  two  cases:  when  the  column  factor  is  fixed  at  its  lowest 
value,  and  when  the  column  factor  is  fixed  at  its  highest  value.  A  quadratic  effect  in  the 
model  is  shown  by  a  curved  line.  When  there  are  no  interactions  between  the  row  and 
column  factors  the  lines  are  dashed  instead  of  being  solid  and  always  parallel.  There  are 
two  lines  because  changing  the  value  of  one  of  the  factors  from  its  low  value  to  its  high 
value  changes  the  MOE. 
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Eigure  13.  Matrix  of  Interaction  Terms  in  Einal  Model  (Scenario-W) 


The  first  interaction  to  consider  is  that  between  USV  speed  and  the  number  of 
USVs.  The  interaction  matrix  shows  that  the  mean  proportion  of  enemies  detected 
increases  as  the  number  of  USV  increases  when  the  enemy’s  speed  is  high.  Eigure  14 
shows  a  contour  plot  of  these  two  factors,  which  verifies  the  reaction  of  the  response 
variable.  The  proportion  of  enemies  that  are  identified  increases  as  the  speed  of  the 
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enemy  increases  and  as  the  number  of  USVs  increases.  Further  clarification  on  the 
tactical  meaning  is  processed  through  the  prediction  profiler  in  JMP™. 
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Figure  14.  Contour  Plot  of  USV  Speed  vs.  the  Number  of  USVs  (Scenario-W) 

jMpTM  ^  Prediction  Profiler  that  (like  the  Interaction  Matrix  graph) 

enables  review  of  the  predicted  regression  model  instead  of  the  original  simulation  output 
data.  The  vertical  dotted  lines  correspond  to  the  levels  of  the  five  factors.  The  horizontal 
dotted  line  corresponds  to  the  predicted  proportion  of  enemies  killed.  The  solid  line  for  a 
particular  sub-graph  shows  the  impact  of  changing  that  factor  from  its  lowest  to  its 
highest  level,  while  the  other  factors  are  held  constant  at  the  levels  by  the  dashed  vertical 
lines  and  numerical  value  below  the  plot.  Figure  15  shows  the  base  case  in  the  profiler 
and,  based  on  the  regression  model,  generates  a  95%  confidence  interval  (Cl)  of  the  mean 
proportion  of  enemy  agents  detected.  Curved  lines  in  the  profiler  display  the  quadratic 
effects  of  the  factors  that  appear  as  squared  terms  in  the  model. 

The  profiler  also  displays  the  relationship  between  each  factor  and  the  predicted 
value  of  the  MOE.  If  the  term  returns  a  flat  line,  this  denotes  a  strictly  linear  relationship 
and  the  term  only  appears  as  a  linear  effect  in  the  model.  If  the  profiler  shows  curvature, 
then  the  term  is  modeled  with  a  quadratic  effect.  Some  quadratic  terms  in  this  model 
have  diminishing  return,  and  others  (in  fact)  show  diminishing  performance.  For 
instance,  in  the  Waypoint  scenario,  the  value  where  the  number  of  USVs  does  not 
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increase  the  proportion  of  deteetions  is  at  17.7  in  the  base  case.  The  interaction  between 
the  USV  speed  and  the  number  of  US  Vs  was  aheady  explored  through  the  prediction 
profiler. 

When  inereasing  both  the  USV  speed  and  number  of  USVs  until  the  MOE  begins 
to  deerease,  while  keeping  the  other  factors  at  their  base  case  levels  (Figure  16),  the  point 
estimate  of  the  prediction  increases  from  0.5652  to  0.6125  and  the  interval  remains 
approximately  the  same  width.  Deereasing  the  USV  speed  to  its  minimum  value  and 
keeping  the  other  variables  constant  (Figure  17)  reduees  the  proportion  of  detections  to 
0.4599,  but  the  interval  widens.  Figure  17  also  shows  that  no  more  than  15.5  USVs  are 
required  when  they  are  moving  at  slower  speeds.  When  the  USVs  operate  at  maximum 
speed,  at  most  20.3  USVs  are  required.  Overall,  this  interaction  suggests  that  18.6, 
rounded  up  to  the  next  nearest  whole  number,  19,  USVs  are  required  when  the  USV 
speed  is  at  its  optimal  value  of  26.7  knots. 


Speed:  Speed:  Enemy  Number  Camera  Simulation 

USV(s)  (knots)  (knots)  USV(s)  Range  (nm)  Length  (minutes) 


Figure  15.  Base  Case  of  Final  Model,  95%  Cl  (0.5320,0.5984)  (Seenario-W) 
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Figure  16.  USV  Speed  and  Number  of  USVs  Interaction;  Diminishing 

Returns  (Scenario- W) 
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Speed:  Speed:  Enemy  Number  Camera  Simulation 

USV(s)  (knots)  (knots)  USV(s)  Range  (nm)  Length  (minutes) 


Figure  17.  USV  Speed  and  Number  of  USVs  Interaction;  Low  USV  Speed 

(Scenario-W) 

The  enemy’s  speed,  in  conjunction  with  the  length  of  the  simulation,  has  a 
different  effect  on  the  proportion  of  detections  made  by  the  USVs.  Both  factors  must  be 
near  the  upper  boundary  of  the  range  for  the  proportion  to  be  greater  than  0.750.  This 
can  be  seen  by  looking  at  the  matrix  of  interactions  in  Figure  18.  The  matrix  shows  a 
divergence  in  the  proportion  of  detections  when  either  factor  is  at  its  maximum  value. 
The  contour  plot  in  Figure  18  also  shows  this  increasing  relationship.  Utilizing  the 
profiler  (Figure  19),  the  analysis  of  the  enemy  speed  and  USV  endurance  interaction 
shows  that  when  the  enemy  speed  is  near  its  minimum  value,  the  optimal  time  on  station 
for  the  USV  is  437  minutes.  The  proportion  of  detections  decreases  (slightly)  with  a 
longer  time  on  station.  The  profile  plot  in  Figure  20  shows  that  when  the  speed  of  the 
enemy  is  near  its  maximum,  a  longer  time  on  station  increases  the  proportion  of 
detections,  although  there  are  diminishing  returns  for  the  simulation  length.  If  the 
simulation  was  permitted  to  run  for  longer  periods  of  time,  there  may  be  a  turning  point 
where  there  is  a  maximum  return  before  the  performance  measure  would  decrease.  This 
cannot  be  said  with  certainty,  as  the  range  of  values  examined  does  not  demonstrate  the 
outcome. 
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Figure  18.  Contour  Plot  for  Enemy  Speed  vs.  Simulation  Length  (Seenario-W) 


Speed:  Speed:  Enemy  Number  Camera  Simulation 

USV(s)  (knots)  (knots)  USV(s)  Range  (nm)  Length  (minutes) 


Figure  19.  Enemy  Speed  and  Simulation  Length  Interaetion;  Low  Enemy  Speed, 
Diminishing  Return  of  Simulation  Length  (Seenario-W) 


Speed:  Speed:  Enemy  Number  Camera  Simulation 

USV(s)  (knots)  (knots)  USV(s)  Range  (nm)  Length  (minutes) 


Ligure  20.  Enemy  Speed  and  Simulation  Length  Interaction;  High  Enemy  Speed, 
Increasing  MOL  in  Simulation  Length  Range  (Scenario-W) 
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There  are  four  quadratie  terms  in  the  final  regression  model.  The  profiler  depiets 
an  individual  “optimal”  point  for  each  quadratic  effect  compared  to  the  base  case. 
Because  they  are  interactions,  the  terms  cannot  be  combined  to  identify  global  optima. 
For  the  USV  speed,  the  value  is  29.0  knots;  more  than  18.1  USVs  decreases  the 
proportion  of  detections;  and,  the  maximum  needed  camera  range  for  the  base  case  is 
7.41  nm.  Time  on  station  does  not  span  the  diminishing  return  point  for  the  base  case. 
Figure  21  shows  the  prediction  profiler  with  the  combination  of  each  of  these  points  and 
the  enemy  speed  at  its  base  case  value.  The  95%  Cl  of  the  proportion  of  detections  is 
(0.7095,  0.7721). 
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Speed:  Speed:  Enemy  Number  Camera  Simulation 

USV(s)  (knots)  (knots)  USV(s)  Range  (nm)  Length  (minutes) 


Figure  21 .  Quadratic  Effects  Against  Base  Case  (Scenario-W) 

2,  Scenario-I  Analysis 

As  with  the  analysis  of  the  Waypoint  scenario,  we  begin  by  constructing  a  model 
which  only  contains  terms  associated  with  factors  the  HVU  can  control — including  the 
main  effects,  the  two-way  interactions,  and  the  quadratic  effects.  Figure  22  shows  that 
the  model  has  a  good  fit,  that  the  regression  is  statistically  significant,  and  that  83.99%  of 
the  variance  is  explained  by  11  significant  factors  that  are  controllable  by  the  HVU. 
However,  the  residual  plot  (Figure  23)  shows  several  outliers,  so  the  model  with  these 
terms  is  inadequate  and  requires  an  alternative.  Since  the  range  of  the  MOE  is  bounded, 
all  points  in  the  residual  plot  must  fall  within  a  diagonal  band. 


39 


Figure  22.  Actual  vs.  Predicted  Responses  for  Significant  Controllable  Factors 

Model  (Scenario-I) 
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Figure  23.  Residual  vs.  Predicted  Responses  for  Significant  Controllable  Factors 

Model  (Scenario-I) 


Considering  all  65  factors,  the  full  model  explains  93.81%  of  the  variance  of  the 
mean  proportion  of  enemy  detections.  The  regression  is  significant  and  the  residual  plot 
shows  neither  non-constant  variance  nor  outliers.  Since  65  terms  are  too  many  for 
decision-making  purposes,  we  again  use  the  stepwise  regression  technique.  The  stepped 
model  has  a  good  fit  (Figure  24)  and  the  residual  plot  (Figure  25)  shows  that  the  model  is 
adequate  with  no  major  defects. 
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Figure  24.  Actual  vs.  Predicted  Responses  for  Stepped  Model  (Scenario-I) 
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Figure  25.  Residual  vs.  Predicted  Responses  for  Stepped  Model  (Scenario-I) 

Even  though  the  plot  shows  a  good  fit,  this  is  a  complicated  model  with  30 
factors.  As  in  the  analysis  of  the  Waypoint  scenario,  we  manually  simplified  the  model 
by  removing  the  terms  that  had  a  p-value  between  0.05  and  0.10.  The  final  model  has 
only  12  terms  and  still  explains  a  considerable  amount,  85.27%,  of  the  variance  of  the 
mean  proportion  of  enemies  killed.  Figure  26  shows  the  model  has  a  good  fit.  Although 
the  residual  plot  (Figure  27)  shows  some  slight  departures  from  the  desired  horizontal 
band,  its  cloud-like  shape  (with  the  highest  scatter  near  a  prediction  of  0.5)  is 
characteristic  of  having  a  response  variable  that  is  a  proportion  between  0  and  1 
(Montgomery,  2001). 

Another  comment  on  the  residual  plot  of  the  final  model  is  that  there  seems  to  be 
a  linear  boundary  of  the  points  in  Figure  27.  This  is  also  a  function  of  the  actual  response 
variable  being  restricted  to  the  interval  [0,1].  For  example,  if  the  model  prediction  is  0.0 
then  the  residual  must  be  positive;  if  the  model  prediction  is  0.1  then  the  residual  cannot 
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be  less  than  -0.1,  ete.  Fortunately,  regression  ean  still  be  used  to  identify  important  terms 
as  it  is  known  to  be  a  good,  unbiased  predietor  of  the  mean  response  even  if  the  varianee 
is  not  constant. 

A  pattern  in  the  residual  plot  would  indicate  invalid  prediction  intervals  for 
individual  responses.  Since  we  are  only  looking  at  identifying  important  terms  and  not 
predicting  individual  responses,  this  slight  variation  in  the  look  of  the  residual  plot  can  be 
overridden.  The  final  model  only  contains  one  more  factor  than  the  model  with 
significant  controllable  factors  and  increases  the  amount  of  variability  that  is  explained 
by  1.28%.  The  added  term  is  the  quadratic  effect  of  the  number  of  USVs.  Table  5 
displays  the  coefficients  that  are  included  in  the  final  model  for  Scenario-I. 


Figure  26.  Actual  vs.  Predicted  Responses  for  Final  Model  (Scenario-I) 
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Figure  27.  Residual  vs.  Predicted  Responses  for  Final  Model  (Scenario-I) 
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Table  5,  Coefficients  in  the  Final  Model  (Scenario-I) 


Term 

Coefficient 

Std  Error 

P-value 

Intercept 

-0.2881184 

0.03972294 

0.0000 

Speed:  USV(s)  (knots) 

0.01227101 

0.00074078 

0.0000 

Number  USV(s) 

0.00978382 

0.0012213 

0.0000 

Range  from  HVU  (nm) 

0.00639677 

0.00147981 

0.0000 

Camera  Range  (nm) 

0.01912085 

0.00311894 

0.0000 

Simulation  Length  (minutes) 

0.00061956 

0.00004935 

0.0000 

(Speed:  USV(s)  -21.104)*(Camera  Range  -5.46848) 

0.00118821 

0.00027425 

0.0000 

(Speed:  USV(s)-21.104)*(Simulation  Length  -317.338) 

0.00001146 

0.00000456 

0.0135 

(Range  from  HVU-10.3578)*(Simulation  Length  -3 17.338) 

0.00002573 

0.00000911 

0.0056 

(Camera  Range  -5.46848)*(Simulation  Length  -317.338) 

0.00004923 

0.00002039 

0.0174 

(Number  USV(s)  -12.528)*(Number  USV(s)  -12.528) 

-0.0006928 

0.00022288 

0.0024 

(Range  from  HVU  -10.3578)*(Range  from  HVU  -10.3578) 

-0.0008192 

3.00E-04 

0.0074 

(Simulation  Length  -317.338)*(Simulation  Length  -317.338) 

-0.0000014 

3.63E-07 

0.0002 

The  final  model  includes  four  interactions  terms.  The  relationships  between  the 
crossed  terms  are  shown  graphically  in  Figure  28.  Each  interaction  is  examined  more 
closely  by  using  the  contour  plots  and  the  profiler  as  in  the  Waypoint  scenario  analysis. 
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Figure  28. 


Matrix  of  Interaction  Terms  in  Final  Model  (Scenario-I) 
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The  significant  interactions  are  those  between  the  USV  speed  and  camera  range; 
USV  speed  and  USV  time  on  station;  combat  radius  (permissive  range  from  HVU);  and 
USV  time  on  station;  and  camera  range  and  time  on  station.  From  the  interaction  matrix, 
there  is  an  obvious  change  in  the  slope  of  the  response  in  the  first  interaction,  USV  speed 
and  camera  range.  Figure  29  demonstrates  strong  positive  main  effects:  as  either  factor 
increases,  so  does  the  proportion  of  enemies  detected.  When  a  sensor  has  a  longer  range, 
it  is  logical  that  it  would  result  in  more  detections. 

Figure  29  shows  that  the  camera  range  does  not  matter  at  low  speeds,  but 
increasing  the  camera  range  is  very  beneficial  at  high  speeds.  In  reality,  this  is  not  to  be 
expected,  unfortunately.  With  increases  in  speed,  platform  stability  decreases  and  so  do 
sensor  ranges.  A  moderate  increase  in  the  speed  has  a  small  impact  on  the  proportion  of 
detections  at  low  ranges,  but  a  much  larger  impact  at  high  ranges.  Figure  29  also  shows 
that  both  factors  need  to  be  at  the  higher  ends  of  their  ranges  in  order  for  the  proportion 
of  detections  to  increase. 

There  is  a  minimum  speed  at  which  the  USVs  should  travel,  in  a  range  of  10-20 
knots.  If  the  speed  is  too  slow,  then  not  even  an  increase  in  the  camera  range  would  raise 
the  proportion  of  detections.  This  makes  tactical  sense. 


I  I  “ 


Speed:  USV(s)  (knots) 

Figure  29.  Contour  Plot  of  USV  Speed  vs.  Camera  Range  (Scenario-I) 

The  interaction  between  the  USV  speed  and  the  simulation  length  does  not  show 
as  drastic  changes  in  the  slopes  as  in  Figure  28,  but  Figure  30  does  show  how  the 
increasing  relationship  between  the  two  factors  jointly  increases  the  proportion  of 
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enemies  detected  in  the  scenario.  This  relationship  is  intuitive  since  the  longer  the  time 
available  for  a  sensor  to  seek  contacts,  the  greater  the  number  of  detections  that  can 
occur. 
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Figure  30.  Contour  Plot  of  USV  Speed  vs.  Simulation  Length  (Scenario-I) 

Figure  31  displays  the  relationship  of  the  proportion  of  detected  enemy  agents 
with  the  interaction  of  the  camera  speed  and  the  simulation  length.  This  relationship  is 
similar  to  the  previous  two.  It  takes  high  levels  of  both  factors  to  achieve  a  high 
proportion  of  detections.  The  contour  plot  shows  that  long  times  on  station  with  a  short 
camera  ranges  result  in  lower  proportions  of  detections  than  short  endurances  and  high 
camera  ranges.  The  US  Vs  should  be  deployed  for  longer  time  periods  only  if  the  camera 
range  can  be  increased  as  well.  If  the  time  on  station  is  longer  with  a  short  camera  range, 
the  proportion  of  detections  is  less  than  if  a  USV  with  the  same  camera  range  were 
deployed  for  a  short  period  of  time. 
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Figure  3 1 .  Contour  Plot  of  Camera  Range  vs.  Simulation  Length  (Seenario-I) 


The  last  interaction  term  in  the  final  model  of  the  Interceptor  scenario  is  the 
combination  of  USV  combat  radius  and  the  time  on  station.  The  contour  plot  (Figure  32) 
shows  some  interesting  effects  of  the  relation  of  the  factors.  As  the  US  Vs  are  permitted 
to  range  further  from  the  HVU,  the  proportion  of  detections  increases,  which  is  intuitive. 
As  the  area  that  is  being  searched  increases,  the  USVs  are  able  to  approach  more 
contacts,  providing  a  larger  proportion  of  enemies  that  are  attainable.  When  the 
simulation  length  and  the  range  are  both  at  their  maximum  values,  the  proportion  of 
detections  is  greater  than  0.750,  which  is  good.  What  is  interesting  is  that  the  proportion 
of  detections  does  not  seem  to  depend  on  the  simulation  length  when  the  range  is  at  its 
lower  limit  but  it  does  when  the  range  is  at  its  upper  limit. 

The  prediction  profiler  (Figure  33)  shows  that  a  short  time  on  station  requires  a 
combat  radius  from  the  HVU  of  9.5  nm.  With  longer  time  on  station,  the  combat  radius 
at  the  point  of  maximum  return  is  18.7  nm.  This  indicates  an  increasing  relationship 
between  the  MOE  and  this  interaction.  A  combat  radius  of  less  than  2.0  nm,  however, 
returns  a  maximum  for  time  on  station  at  455  minutes  (Figure  34). 
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Figure  32.  Contour  Plot  of  Permissive  Range  vs.  Simulation  Length  (Seenario-I) 


Speed:  Number  Range  from  Camera  Simulation 

USV(s)  (knots)  USV(s)  HVU  (nm)  Range  (nm)  Length  (minutes) 


Figure  33.  Base  Case  Final  Model  95%  Cl  (0.4226,  0.4948)  (Scenario-I) 
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HVU  (nm)  Range  (nm)  Length  (minutes) 


Figure  34.  Short  Time  on  Station;  Maximum  Return  of  Permissive  Range 

(Seenario-I) 
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Speed:  Number  Range  from  Camera  Simulation 

USV(s)  (knots)  USV(s)  HVU  (nm)  Range  (nm)  Length  (minutes) 

Figure  35.  Short  Permissive  Range;  Maximum  Return  of  Time  on  Station 

(Scenario-I) 

Using  the  predietion  profiler  to  look  at  the  quadratic  effects  of  the  number  of 
USV,  combat  radius,  and  simulation  length,  the  point  of  maximum  return  for  each  of 
these  factors  can  be  individually  compared  to  the  base  case.  Figure  36  demonstrates  the 
points  with  USV  speed  and  camera  range  kept  at  the  base  case  values.  The  point  of 
maximum  return  occurs  with  19.5  USVs,  a  combat  radius  of  15.4  nm,  and  time  on  station 
of  535  minutes.  The  95%  Cl  for  these  points  and  the  other  factors  at  their  base  case  is 
(0.5615,  6497). 


Speed:  Number  Range  from  Camera  Simulation 

USV(s)  (knots)  USV(s)  HVU  (nm)  Range  (nm)  Length  (minutes) 


Figure  36.  Quadratic  Effects  Against  Base  Case  (Scenario-I) 

3,  Comparison  between  Scenario-W  and  Scenario-I 

When  looking  at  the  two  final  regression  models  for  each  of  the  ISR  scenarios, 
there  are  some  factors  that  are  significant  in  both  and  some  that  are  only  significant  in 
one  model.  Table  6  decomposes  the  terms  in  each  model.  Terms  that  are  horizontally 
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aligned  and  in  bold  appear  in  both  models.  It  is  interesting  that  these  terms  are  all 
eontrollable.  We  briefly  eompare  and  eontrast  the  impaet  of  these  eommon  terms. 


Table  6,  Side-by-side  Comparison  of  the  Factors  in  the  Waypoint  and 

Interceptor  Regression  Models 


Waypoint  Model 

Interceptor  Model 

Speed:  USV(s) 

Speed:  USV(s) 

Speed:  Enemy 

Number  USV(s) 

Number  USV(s) 

Number  of  Contacts 

USV  Range  from  HVU 

Camera  Range 

Camera  Range 

Simulation  Length 

Simulation  Length 

Speed:  USV(s)*Number  USV(s) 

Speed:  USV(s)*Camera  Range 

Speed:  USV(s)*Simulation  Length 

Speed:  Enemy*Simulation  Length 

Camera  Range*Simulation  Length 

USV  Range  from  HVU*Simulation  Length 

Speed:  USV(s)  *Speed:  USV(s) 

Number  USV(s)*Number  USV(s) 

Number  USV(s)*Number  USV(s) 

USV  Range  from  HVU*USV  Range  from  HVU 

Camera  Range  *Camera  Range 

Simulation  Length*Simulation  Length 

Simulation  Length*Simulation  Length 

We  first  take  a  eloser  look  at  the  faetors  that  are  eommon  to  both  regression 
models.  USV  speed  appears  as  a  main  effeet  with  a  positive  eoeffieient  in  both  models, 
although  its  eoeffieient  in  the  Waypoint  model  is  mueh  smaller  than  that  in  the 
Intereeptor  model  (0.0021  vs.  0.0123).  This  suggests  that  high  USV  speeds  are  less 
benefieial  when  the  USVs  travel  predetermined  paths.  The  negative  eoeffieient  for  the 
quadratie  effeet  of  USV  speed  in  the  Waypoint  model  indieates  that  inereasing  the  speed 
has  diminishing  returns,  and  may  eventually  be  eounterproduetive,  ereating  holes  in 
eoverage.  Finally,  USV  speed  appears  in  interaetions  in  both  models,  although  the 
interaetions  involve  different  (but  eontrollable)  faetors.  This  suggests  that  any  “optimal” 
USV  speed  would  depend  on  other  eharaeteristies  of  USV  deployment.  Inereasing  the 
USV  speed  inereases  the  proportion  of  deteetions  experiment  for  the  majority  of  oases 
examined. 
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The  number  of  USVs  appears  with  a  positive  main  effeet  and  a  negative  quadratie 
effect  in  each  regression  model.  This  translates  to  a  diminishing  return  from  increasing 
the  number  of  USVs  in  each  model.  Since  there  is  an  interaction  with  the  USV  speed  in 
Scenario- W,  a  range  of  the  value  to  achieve  maximum  returns  for  the  number  of  USV  is 
between  16  and  19  USVs  when  the  speed  is  at  its  minimum  and  maximum  values, 
respectively.  There  is  no  interaction  for  Scenario-I;  the  value  where  the  maximum  return 
occurs  is  20  USVs.  According  to  our  models,  having  larger  numbers  of  USVs  would 
decrease  the  proportion  of  detections;  though  over  the  factor  ranges  investigated  these 
decreases  are  minimal  (see  Figures  17  and  36). 

Finally,  the  last  main  effect  that  occurs  in  both  regression  models  is  the  simulation 
length.  The  factor  also  appears  as  a  quadratic  term  in  each  of  the  models.  The  maximum 
time  on  station  before  the  performance  of  the  MOE  decreases  is  near  the  maximum  in  the 
range,  600  minutes  for  the  Waypoint  model  and  around  475  minutes  in  the  Interceptor 
model.  The  simulation  length  appears  as  an  interaction  with  the  enemy  speed  in  the 
Waypoint  model  and  with  the  USV  speed,  camera  range  and  permissive  range  in  the 
Interceptor  model.  The  interaction  in  the  Waypoint  model  is  positively  related  to  the 
factors  and  the  MOE. 

When  both  factors  are  increased,  the  MOE  increases  even  more  than  the  main 
effects  would  suggest.  The  range  of  the  simulation  length  based  on  the  enemy  speed  is 
420  -  600  minutes.  Ten  hours  is  the  upper  limit  since  it  is  the  maximum  value  of  the 
simulation  length  in  this  experiment.  Eor  Scenario-I,  the  best  interval  for  time  on  station 
should  be  Scenario-I  is  455  -  600  minutes.  The  intersection  of  these  two  ranges,  455  - 
600  minutes,  is  the  suggested  range  for  the  time  on  station  for  the  USV. 

As  for  the  factors  that  are  different,  the  enemy’s  speed  is  important  in  the 
Waypoint  model  as  a  main  effect  and  an  interaction,  but  does  not  appear  in  any  way  in 
the  analysis  of  the  Interceptor  model.  This  implies  that  the  knowledge  of  the  enemy  is 
more  important  in  the  Waypoint  scenario  than  in  the  Interceptor  scenario.  However,  not 
knowing  the  enemy’s  speed  does  not  appear  to  change  the  operational  tactics  suggested 
by  the  Interceptor  model. 
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Another  difference  is  the  permissive  range  from  the  HVU,  which  appears  in 
several  terms  in  the  regression  model  for  the  Interceptor  analysis  but  does  not  appear  in 
the  Waypoint  model.  Again,  this  is  logical  since  the  USVs  remain  on  a  predetermined 
path,  which  is  constant  throughout  the  experimentation,  in  the  Waypoint  scenario.  A 
variation  of  the  Waypoint  scenario  might  be  to  specify  a  path  for  the  USVs  but  allow 
them  to  leave  this  path  if  they  detected  an  enemy,  and  then  return  once  the  enemy  was 
killed.  The  results  for  the  Interceptor  model  suggest  that  the  permissible  range  might  be 
important  in  this  type  of  scenario.  This  is  a  potential  area  for  future  research. 

The  camera  range  shows  up  as  a  main  effect  in  each  regression  model  but  only  as 
a  quadratic  term  in  the  Waypoint  regression  model.  Again,  intuition  suggests  that  if  the 
USVs  are  more  restricted  in  their  movement,  then  the  maximum  range  of  their  sensor 
plays  a  more  substantial  role  in  the  proportion  of  detections.  The  quadratic  effect  shows 
that  after  some  distance,  the  performance  measure  decreases.  Looking  at  Figure  17,  there 
is  a  maximum  return  point;  that  distance  is  at  7.41  nm.  There  is  an  interaction  between 
the  camera  range  and  the  USV  speed  in  the  Interceptor  regression  model.  For  the  lowest 
speed,  the  camera  range  has  little  impact  on  the  proportion  of  detections.  At  high  speeds, 
a  longer  camera  range  increases  the  proportion  of  detections. 

Overall,  there  is  a  point  where  too  many  USVs  can  cause  a  decrease  in  the 
proportion  of  detections.  USV  speed  in  Scenario-W  does  not  need  to  be  maximized  but  it 
should  be  in  a  scenario  that  closely  represents  Scenario-I.  In  the  Waypoint  model,  even 
though  more  enemies  were  detected  using  a  high  sensor  range,  there  is  a  point  that  the 
camera  range  does  not  need  to  exceed.  Conversely,  the  analysis  of  Scenario-I  shows  that 
as  long  as  the  camera  range  increases,  so  will  detections.  Finally,  time  on  station  in  each 
model  shows  the  USV  endurance  that  is  needed  for  each  scenario  given  that  the  number 
of  enemies  started  with  does  not  change. 

4,  Scenario-FP  Analysis 

While  the  Force  Protection  scenario  contains  the  same  MOE  as  the  ISR  scenarios, 
with  the  threat  agents  included,  two  additional  MOEs  are  considered.  The  two 
accompanying  MOEs  are  the  proportion  of  the  threatening  enemy  agents  that  are  detected 
and  the  number  of  the  threatening  enemy  agents  that  reach  the  HVU  before  being 

detected.  The  description  of  the  analysis  is  in  that  order. 
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Before  any  further  analysis,  it  is  once  more  necessary  to  translate 
PYTHAGORAS  representations  to  tactical  definitions.  During  the  process  of  obtaining 
FP  data,  it  turned  out  that  threat  density  among  contacts  ranging  from  10-90%  was  not 
achieved  properly,  in  part  because  of  the  restriction  of  factor  levels  to  integer  values. 
Instead,  the  actual  range  turns  out  to  be  0-100%.  Overall,  a  total  of  125  design  points 
were  used  with  30  replications  at  each  design  point  for  a  total  of  3750  simulation  runs. 
Because  the  final  design  matrix  had  correlations  less  than  |0.08|,  multicollinearity  is  not  a 
concern  in  the  model-building  steps. 

a.  Proportion  of  Enemies  Detected  Analysis 

In  an  attempt  to  have  a  similar  approach  to  analysis  as  in  the  previous 
cases,  we  began  by  investigating  the  overall  proportion  of  detections  of  both  types  of 
enemy  agents,  threatening  and  non-threatening.  An  initial  look  at  the  regression 
involving  only  the  significant  controllable  factors,  the  quadratic  terms  and  the  two-way 
interactions  is  statistically  significant  (p-value<0.001)  but  explains  only  19.39%  of  the 
variance  in  the  MOE.  Next,  we  used  stepwise  regression  where  all  77  terms  (main 
effects,  quadratic  effects,  and  two-way  interactions)  were  potential  explanatory  variables. 
The  resulting  model  is  significant  (p-value<0.001),  but  the  10  terms  explain  only  31.99% 
of  the  variance.  Figure  37  clearly  shows  a  deviance  of  a  “good”  fit  when  compared  to  the 
Actual  vs.  Predicted  plots  from  previous  analyses.  The  residual  plot  (Figure  38)  shows  a 
weak  fit,  but  does  not  reveal  any  patterns  or  outliers  that  would  suggest  ways  the  model 
could  be  improved. 


Figure  37.  Actual  vs.  Predicted  Responses  for  Stepped  Model  (Scenario-FP) 
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Figure  38.  Residual  vs.  Predieted  Responses  for  Stepped  Model  (Seenario-FP) 


Since  regression  does  not  seem  to  predict  well  for  the  combination  of 
these  FP  factors  and  the  MOE,  next  we  look  at  using  a  nonparametric  partitioning 
approach,  called  regression  trees.  This  methodology  helps  identify  which  factors  can  be 
split,  and  at  what  levels,  in  order  to  cluster  the  data  into  groups.  Ideally,  the  points  within 
a  group  will  have  similar  MOE  values,  but  the  MOE  values  will  differ  widely  across 
groups.  Candidate  factors  have  large  sum-of-squares  values.  The  first  factor  that  can  be 
split  at  a  distinct  level  is  the  percentage  of  threatening  enemies  (Eigure  39).  This  means 
that  if  the  data  points  are  split  into  two  groups,  one  where  the  percentage  of  threatening 
enemies  is  less  than  17  and  the  other  where  the  percentage  is  greater  than  or  equal  to  17, 
the  mean  of  the  proportion  of  detections  would  be  0.4409  and  0.7227,  respectively.  The 
first  split  gives  an  R-squared  value  of  0.138  (13.8%)  with  only  one  term. 


Eigure  39.  Eirst  Split  of  Regression  Tree  in  the  Overall  Proportion  of  Enemy 

Detections  (Scenario-EP) 
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Subsequent  splits  ean  continue  until  any  stopping  criterion  is  met. 
Multiple  splits  can  be  made  on  the  same  factor.  After  5  factors  are  shown  to  explain  the 
performance  of  the  MOE,  a  total  of  5  splits  are  required,  the  second  and  third  illustrated 
in  Figure  40.  The  second  split  occurs  when  the  percentage  of  threatening  enemies  is 
greater  than  17  and  the  new  split  occurs  when  the  speed  of  the  USV  at  28  knots.  A 
higher  speed  with  a  higher  percentage  of  threatening  enemies  gives  a  mean  of  0.7955  out 
of  42  observations.  Likewise,  an  explanation  can  be  stated  for  each  successive  split. 


Figure  40.  Second  and  Third  Splits  in  a  Regression  Tree  (Scenario-FP) 


A  summary  of  the  splits  appears  in  the  Leaf  Table  (Table  7).  An  example 
of  how  to  read  the  Leaf  Table  for  the  highest  mean  (0.8241  in  the  last  row)  follows:  if 
the  percentage  of  threatening  enemies  is  greater  than  17%  and  the  USV  speed  is  greater 
than  28  knots  and  the  range  from  HVU  is  greater  than  3.96  nm,  then  the  mean  overall 
proportion  of  detections  of  enemy  agents  is  0.8241.  This  leaf  contains  37  out  of  the  125 
total  observations. 
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Table  7,  Leaf  Table:  Overall  Proportion  of  Enemy  Contacts  Detected 


(Scenario-FP) 


Leaf 

Mean 

Count 

%  Enemies  Threatening<17&Range  IfomHVU  (nm)>=12.28 

0.27492 

5 

%  Enemies  Threatening<17&Range  IfomHVU  (nm)<12.28 

0.55943 

7 

%  Enemies  Threatening>=17&Speed;  USV(s) 
(knots)<28&Number  USV(s)  <3 

0.36525 

5 

%  Enemies  Threatening>=17&Speed:  USV(s) 
(knots)<28&Number  USV(s)  >=3 

0.70354 

66 

%  Enemies  Threatening>=17&Speed:  USV(s) 
(knots)>=28&Range  Ifom  HVU  (nm)<3.96 

0.58318 

5 

%  Enemies  Threatening>=17&Speed:  USV(s) 
(knots)>=28&Range  IfomHVU  (nm)>=3.96 

0.82416 

37 

Another  summary  of  the  faetors  used  is  the  Factor  Contribution  Chart  (Figure  41) 
which  displays  the  sum-of-squares  value  for  the  main  effects  of  each  factor.  Table  7  and 
Figure  41  both  summarize  the  performance  of  the  partitions.  The  final  R-squared  value 
after  5  partitions  is  0.358.  Therefore,  only  looking  at  5  main  effects,  35.8%  of  the 
variance  can  be  explained.  The  regression  tree  results  are  worth  much  more  than  the 
regressions  that  are  not  significant  or  regressions  where  less  variance  is  explained  with  5 
additional  terms. 


Factor  Contributions 


Camera  Range  (nm) 
Range  from  HVU  (nm) 
%  Enemies  Threatening 
%  Contacts  Enemies 
Totai  Number  Contacts 
Number  USV{s) 
Sea  State 

Speed:  Non-threatening  Enemies 
Speed:  Threatening  Enemies  (knots) 
Speed:  Neutrais 
Speed:  USV(s)  (knots) 


0  0.2  0.4  0.6  0.8 


1.2 


Figure  41 .  Contribution  of  Each  Factor  in  the  Overall  Proportion  of  Enemies 

Regression  Tree  (Scenario-EP) 
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The  most  important  factor  in  explaining  the  variance  of  the  overall 
proportion  of  enemies  is  the  threat  density.  This  indicates  that  knowledge  about  the 
enemy  is  more  important  in  the  FP  scenario  than  in  the  Waypoint  and  Interceptor 
scenarios.  When  the  combat  radius  is  split,  it  suggests  that  large  ranges  are  good  when 
there  is  a  low  percentage  of  threatening  enemies,  while  low  ranges  are  good  when  the 
percentage  of  threatening  enemies  is  high.  The  split  on  the  number  of  USVs  implies  that 
having  more  than  three  USVs  does  not  seem  to  help  much  in  this  FP  scenario  for  the 
overall  proportion  of  enemies  detected. 

b.  Proportion  of  Threatening  Enemies  Detected  Analysis 
We  next  explore  the  mean  proportion  of  threatening  enemies  detected.  As 
Figures  42  shows,  the  vast  majority  of  design  points  had  perfect  detection  (MOE  =  1.0). 
A  preliminary  test  for  Significance  of  Regression  quickly  shows  that  regression  is  not  an 
appropriate  tool  for  modeling  this  MOE.  Not  one  of  the  controllable  factors  is  significant 
and  including  all  77  factors  does  not  lead  to  a  model  that  provides  a  good  fit  or  one  that 
can  be  used  for  prediction. 


Eigure  42.  Actual  vs.  Predicted  Responses  for  Pull  Model  (Scenario-PP) 

This  MOE  also  requires  the  use  of  non-parametric  methods  for  analysis. 
We  once  again  see  what  insights  can  be  gained  by  using  regression  trees.  The  process  is 
the  same  as  before,  and  summaries  of  the  analysis  are  in  Table  8  and  Eigure  44.  Again, 
the  first  factor  that  is  split  is  the  percentage  of  enemies  that  are  threatening.  This  split 
occurs  at  a  level  of  12%,  and  results  in  explaining  16.8%  of  the  variance  in  the  MOE. 
The  second  split  is  at  the  number  of  USV  factor.  When  there  are  one  or  two  USVs  and 

more  than  12%  of  the  enemies  are  threatening,  the  proportion  of  threatening  enemy 
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detections  is  0.9485.  When  more  than  12%  of  the  enemy  contacts  are  threatening,  having 
three  or  more  USVs  results  in  0.9995  as  the  proportion  of  threatening  enemies  detected. 
The  second  split  increases  the  R-squared  value  to  0.194.  At  the  third  partition,  the  R- 
squared  value  equals  0.215  and  subsequent  splits  do  not  tangibly  increase  the  value. 


Table  8,  Leaf  Table:  Proportion  of  Threatening  Enemies  (Scenario-FP) 


Leaf 

Mean 

Count 

%  Enemies  Threatening<12 

0.8 

5 

%  Enemies  Threatening>=12&Number  USV(s)  <3&Speed:  USV(s) 
(knots)<26 

0.90524147 

6 

%  Enemies  Threatening>=12&Number  USV(s)  <3&Speed:  USV(s) 
(knots)>=26 

0.99175926 

6 

%  Enemies  Threatening>=12&Number  USV(s)  >=3 

0.99950617 

108 

Essentially,  the  mean  proportion  of  detections  decreases  when  threat 
density  is  less  than  12%.  When  more  than  12%  of  the  enemies  are  threatening,  the 
proportion  of  detections  depends  on  the  number  of  USVs  and  the  USV  speed.  The 
highest  proportion  of  detections  occurs  with  three  or  more  USVs.  If  less  than  three  USVs 
are  available,  detections  are  high  if  the  speed  of  the  USVs  is  greater  than  or  equal  to  26 
knots.  If  the  speed  is  less  than  26  knots,  then  the  mean  proportion  of  detections  is  lower. 
Clearly,  slightly  lower  detection  probabilities  may  be  less  of  a  concern  if  the  total  number 
of  threatening  enemies  is  small.  This  leads  to  the  discussion  of  the  next  MOE. 


Factor  Contribution 


Camera  Range  (nm) 

Range  from  HVU  (nm) 

%  Enemies  Threatening 
%  Contacts  Enemies 
Total  Number  Contacts 
Number  USV(s) 

Sea  State 

Speed:  Non-threatening  Enemies 
Speed:  Threatening  Enemies  (knots) 

Speed:  Neutrals 
Speed:  USV(s)  (knots) 

0  0.05  0.1  0.15  0.2 


Eigure  43.  Contribution  of  Each  Eactor  in  Proportion  of  Threatening  Enemies 

Regression  Tree  (Scenario-FP) 
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c.  Number  of  Threatening  Enemies  that  Reach  the  HVU 

The  controllable  factors,  USV  speed,  number  of  USVs,  camera  range  and 
permissive  range  from  HVU,  are  the  first  to  be  looked  at  for  the  regression  against  the 
number  of  threatening  enemies  that  reach  the  HVU.  A  regression  with  the  main  effects 
of  the  controllable  factors,  interactions  and  quadratic  effects  (not  shown)  indicated  a  great 
deal  of  heteroscedacity.  This  suggests  that  a  transformation  of  the  response  variable  is 
appropriate.  Since  the  MOE  is  a  count  which  is  characteristic  of  a  Poisson  distribution,  a 
square-root  transformation  could  make  the  residual  plot  of  the  regression  less 
heteroscedastic.  After  transforming  the  MOE,  the  analysis  continues  with  a  stepwise 
procedure  of  the  controllable  factors.  This  yields  a  0.4853  R-squared  value  with  only  the 
number  of  USVs,  its  quadratic  effect,  and  the  camera  range  in  the  model.  The  actual  vs. 
predicted  plot  (Eigure  44)  does  not  show  a  tight  fit  even  though  it  passes  the  test  for 
significance  of  regression. 


Eigure  44.  Actual  vs.  Predicted  Responses  for  Significant  Controllable  Factors  for 

Number  of  Threatening  Enemies  that  Reach  the  HVU  (Scenario-FP) 

Therefore,  the  inclusion  of  all  of  the  factors’  main  effects,  two-way 
interactions  and  quadratic  effects  are  regressed  on  the  MOE.  Initially,  95.28%  of  the 
variance  is  explained  by  the  77  factors,  but  as  mentioned  previously,  a  model  with  many 
fewer  terms  that  can  still  explain  a  considerable  amount  of  the  variance  is  a  better  option 
to  present  to  decision  makers.  Using  stepwise  regression  followed  by  manual  removal  of 
some  marginally  significant  terms  yields  the  final  model  that  explains  89.72%  of  the 
variance  with  only  12  terms.  Figure  45  shows  the  fit  of  the  final  model  via  the  actual 
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value  plotted  against  the  predicted  values,  and  Figure  46  plots  the  residuals  against  the 
predicted  values.  Once  again,  the  line  on  the  lower  left  of  the  plot  is  representative  of  the 
boundary  that  the  number  cannot  be  lower  than  zero,  since  it  is  a  count.  Of  greater 
concern  is  the  number  of  points  where  the  model  predicts  a  negative  value,  as  well  as  the 
cluster  of  six  points  circled  in  the  upper  right-hand  corner  of  Figure  46.  These  are  related 
since  the  cluster  has  a  strong  influence  on  the  regression  model. 

Further  investigation  showed  that  most  of  these  points  were  associated 
with  a  small  number  of  US  Vs,  but  after  checking  the  original  data  we  found  no  reasons 
for  eliminating  these  points.  The  model  fit  could  be  improved  if  a  dummy  variable 
corresponding  to  US  Vs  less  than  four  were  added  as  an  explanatory  term,  but  the 
problem  of  negative  predictions  remained.  Because  of  this,  we  decided  regression  trees 
were  a  more  appropriate  analysis  tool. 


Figure  45.  Actual  vs.  Predicted  Responses  for  Final  Model  (Scenario-FP) 


Figure  46.  Residual  vs.  Predicted  Responses  for  Final  Model  (Scenario-FP) 
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Consistent  with  the  residual  plot,  the  first  split  in  the  regression  tree  occurs 
at  four  US  Vs  (Figure  47),  with  an  R-squared  value  of  0.369.  This  single  factor  is 
explaining  36.9%  of  the  variance.  The  subsequent  splits  are  shown  in  the  Leaf  Table 
(Table  9).  After  making  five  partitions,  we  find  that  68.1%  of  the  variance  is  explained 
by  only  four  factors.  This  is  our  final  model  since  any  more  splits  increase  the  R-Squared 
value  by  less  than  1%.  Table  9  shows  that  if  the  number  of  USV  is  greater  than  or  equal 
toll,  the  mean  number  of  threatening  enemies  that  reach  the  HVU  is  0.5310.  If  the 
number  is  bound  between  four  and  less  than  1 1 ,  and  the  total  number  of  contacts  is  less 
than  345  of  the  possible  500,  then  the  mean  number  is  1.2372.  Fewer  than  four  USVs 
alone  produced  a  mean  of  8.5044,  but  if  the  percentage  of  enemies  that  are  threatening  is 
examined,  then  the  mean  changes.  If  the  percentage  is  less  than  40%,  then  the  mean  is 
2.5667:  if  the  percentage  of  threatening  enemies  is  greater  than  40%,  the  mean  increases 
greatly  to  13.7  threatening  enemies  who  reach  the  HVU.  Figure  48  is  a  chart  of  the 
relative  contributions  of  each  factor  that  is  varied  throughout  the  simulation. 


Figure  47.  First  Split  for  Number  Threatening  Enemies  that  Reach  HVU 

(Scenario-FP) 
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Table  9,  Leaf  Table:  Number  of  Tbreatening  Enemies  that  Reach  tbe  HVU 

(Scenario-FP) 


Leaf 

Mean 

Count 

Number  USV(s)  >=4&Number  USV(s)  >=11 

0.53102 

72 

Number  USV(s)  >=4&Number  USV(s)  <1  l&Total  Number 
Contacts<345 

1.23718 

26 

Number  USV(s)  >=4&Number  USV(s)  <1  l&Total  Number 
Contacts>=345&Speed:  Threatening  Enemies  (knots)<32 

2.39444 

6 

Number  USV(s)  >=4&Number  USV(s)  <1  l&Total  Number 
Contacts>=345&Speed:  Threatening  Enemies  (knots)>=32 

5.58889 

6 

Number  USV(s)  <4&%  Enemies  Threatening<40 

2.56667 

7 

Number  USV(s)  <4&%  Enemies  Threatening>=40 

13.7 

8 

Factor  Contribution 

Camera  Range  (nm) 
Range  from  HVU  (nm) 

%  Enemies  Threatening 
%  Contacts  Enemies 
Total  Number  Contacts 
Number  USV(s) 
Sea  State 

Speed:  Non-threatening  Enemies 
Speed:  Threatening  Enemies  (knots) 
Speed:  Neutrals 
Speed:  USV(s)  (knots) 

■ 

0  2( 

)0  400  6( 

)0  800  1000 

Figure  48.  Factor  Contribution  Chart:  Number  of  Threatening  Enemies  that  Reach 

the  HVU  (Scenario-FP) 


B.  VERIFICATION  AND  VALIDATION 

There  are  some  analytical  methods  that  can  be  used  to  check  the  validity  of  the 
results  the  simulation  data  gives.  The  probability  of  detection  in  the  Waypoint  scenario 
can  be  compared  to  the  analytical  probability  of  detection  using  the  sweep  width  of  the 
sensor.  The  Interceptor  scenario  has  each  USV  independently  and  individually  choosing 
which  contact  to  search.  By  looking  at  the  analytical  computations,  a  comparison 
between  the  simulation  results  and  the  Random  Search  model  can  confirm  or  refute  the 
simulation  results.  The  computational  formulas  are  from  Naval  Operations  Analysis 
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(Wagner,  et  al.,  1999).  The  Waypoint  and  Intereeptor  scenarios  have  much  greater 
fidelity  and  resolution  than  the  random  search  models.  Even  so,  comparing  the 
simulation  results  with  the  analytical  computations  yields  some  insight  into  the 
simulation  models’  validity  under  specific  conditions. 

First,  consider  comparisons  for  the  Waypoint  scenario.  Sweep  width  (w)  is  the 
area  under  the  static  probability  of  detection  (Pd)  curve  of  the  sensor.  Since  the  USVs  are 
moving,  the  new  Pd  is 

w 

p,=-  (1) 

s 

where  s  is  the  space  between  the  patrol  routes.  For  the  Waypoint  scenario,  s  is  held 
constant  at  100  nm  and  w  changes  as  the  camera  range  factor  varies.  Equation  (1)  shows 
that  when  spacing  is  kept  constant,  the  Pd  increases  as  w  increases.  This  occurs  when  the 
sensor  range  increases  in  the  Waypoint  (and  other)  scenarios,  therefore,  the  Pd  increases 
with  the  sensor  range.  Probabilities  of  detection  for  a  few  combinations  of  sweep  width 
(camera  range)  and  the  space  between  patrols  are  shown  in  Table  10. 

Table  10.  Analytical  Values  for  Scenario-W 


Sweep  Width 

(Camera  Range) 

Min 

Max 

Space  between  patrols 

(constant  at  100  nm) 

0.02 

0.155 

A  scatter  plot  of  the  camera  range  vs.  the  mean  of  simulated  values  at  each  range 
is  provided  in  Figure  49.  The  plot  looks  at  the  mean  of  the  MOE  at  each  level  of  the 
camera  range.  Comparing  these  means  to  the  analytical  values  (Table  10),  there  is 
obviously  not  much  consistency  between  these  values.  Since  there  are  abstractions  to  the 
scenario,  stated  in  Chapter  11,  this  is  one  reason  that  the  simulated  values  do  not  match 
the  analytical  values.  However,  Pd  increases  relatively  linearly  as  the  range  increases, 
which  happens  in  the  analytical  equation.  A  property  of  the  simulated  data  that  is  not 
characteristic  of  the  analytical  model  is  that  the  camera  range  has  a  quadratic  effect  in  the 
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regression  equation  for  Scenario-W.  This  could  be  related  to  the  fidelity  with  which  we 
chose  to  model  the  patrol  patterns. 


Figure  49.  Scatter  Plot  of  Camera  Range  vs.  Mean  Proportion  of  Enemies  Detected 

(Scenario-W) 

Next,  we  consider  the  Interceptor  scenario.  The  random  search  formula  gives  a 
cumulative  probability  Fd(t),  the  probability  that  at  least  one  detection  occurs  in  the  time 
t.  The  range  of  the  sensor  (r),  the  velocity  of  the  search  vehicle  (v),  the  size  of  the  area  to 
be  searched  {A),  and  the  time  that  the  search  is  being  conducted  (t)  are  the  factors  that 
determine  the  Fa.  All  of  these  contributors  are  varied  in  the  simulation  models  except  the 
area,  which  is  kept  at  a  constant  1600  sq-nm.  Using  the  following  equation, 

(2) 

cumulative  probabilities  can  be  determined  for  various  factor  combinations.  As  in  the 
analytical  case  for  the  Waypoint  scenario,  we  computed  cumulative  probabilities  of 
detection  for  combinations  of  the  minimum  and  maximum  factor  values. 


Table  11.  Analytical  Values  for  Scenario-I 


Fd 

Camera  Range/Velocity(Speed) 

Min/Min 

Min/Max 

Max/Min 

Max/Max 

Time 

Min 

0.086 

0.834 

0.593 

1.0 

Max 

0.777 

1.0 

.9999997 

1.0 
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These  analytieal  values  can  be  qualitatively  compared  with  contour  plots  of  the 
simulation  results.  Only  two  factors  can  be  looked  at  simultaneously,  so  a  total  of  three 
contour  plots  are  examined.  The  first  is  the  camera  range  and  speed  contour  plot  (Figure 
50).  The  simulation  results  appear  to  correspond  to  the  Pd  analytical  values  (looking 
across  the  values  in  Table  11),  since  increasing  either  factor  increases  the  Pd  and 
increasing  both  raises  the  proportion  of  detection  even  further.  The  simulation  results 
also  conform  to  the  analytical  results  in  that  the  lower  right  hand  corner  (high  speed,  low 
camera  range)  tends  to  yield  higher  detection  probabilities  than  the  upper  left-hand  corner 
(high  camera  range,  low  speed). 
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Figure  50.  Contour  Plot  of  Camera  Range  and  USV  Speed  (Scenario-I) 


The  analytical  values  for  Scenario-I  show  that  the  Pd  increases  as  the  period  of 
time  spent  searching  increases.  Figures  51  and  52  both  show  this  relationship  between 
the  values  of  the  simulation  length  factor.  Once  again,  increasing  either  the  camera  range 
or  the  USV  speed  increases  the  Pd.  Figure  51  shows  that  the  combination  of  high 
endurance  and  short  camera  range  tend  to  yield  higher  detection  probabilities  than  the 
low  time,  high  camera  range  combination.  These  conform  to  the  analytic  results  when 
the  speed  is  low;  when  the  speed  is  high,  both  combinations  perform  well. 


Mean(Proportion  Enemies  Killed) 

■  <=  0.125 
H  <=  0.250 
H  <=  0.375 

■  <=  0.500 
H  <=  0.625 
H  <=  0.750 
H  >  0.750 
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Figure  5 1 .  Contour  Plot  of  Camera  Range  vs.  Simulation  Length  (Seenario-I) 
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Figure  52.  Contour  Plot  of  USV  Speed  vs.  Simulation  Length  (Seenario-I) 

The  analytieal  values  are  optimistie  when  eompared  to  the  simulation  values 
beeause  both  analytieal  formulas  are  designed  for  statie  targets,  and  the  targets  are 
moving  in  eaeh  of  the  three  seenarios  in  this  thesis. 

C.  SCENARIO  COMPARISONS  AND  INSIGHTS 

We  have  examined  and  eompared  results  for  five  situations  involving  three  MOEs 
(proportion  of  enemy  deteetions,  proportion  of  threatening  enemies  deteeted,  number  of 
threatening  enemies  that  reaeh  the  FIVU)  and  three  seenarios  (Waypoint,  Intereeptor,  and 
Force  Protection).  Although  two  different  analysis  approaches  were  used,  we  can 
compare  the  results  across  scenarios  by  looking  at  the  common  terms  (Table  12).  The 
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shaded  cells  show  where  the  final  models  do  not  contain  the  specific  factor.  One  factor  is 
significant  in  each  of  the  five  analyses — the  number  of  USVs.  USV  speed  is  significant 
in  all  analyses  except  the  FP-number  that  reach  HVU.  Camera  range  and  simulation 
length  are  important  in  the  two  regression  analyses  discussed  in  the  comparison  of  the 
two  models.  Finally,  the  percentage  of  threatening  enemies  is  important  in  each  of  the  FP 
analyses,  which  were  the  only  analyses  to  consider  this  factor. 

The  good  news  for  the  planners  and  decision  makers  is  that  these  are  all 
controllable  factors  except  the  percentage  of  threatening  enemies.  This  means  that 
decisions  to  deploy  the  USV  in  one  of  the  three  proposed  scenarios  are  subject  to  neither 
inaccurate  information  nor  assumptions  in  the  face  of  uncertainty.  This  is  not  to  say  that 
the  models’  other  significant  factors  are  trivial,  but  regardless  whether  tasking  involves 
Interceptor  or  Force  Protection  missions,  some  important  information  is  already  known 
about  the  impact  of  the  number  of  USVs,  the  USV  speed,  and  the  sensor  range. 


Table  12.  Comparison  of  Model  Terms 


Model-W 

Model-I 

Model-FP 

Proportion 
of  Enemies 

Deteeted 

Proportion 

of 

Enemies 

Deteeted 

Overall 
Proportion 
of  Enemies 

Deteeted 

Proportion  of 
Threatening 
Enemies 

Deteeted 

Number  of 
Threatening 
Enemies  that 

Reaeh  the  HVU 

USV(s)  speed 

X 

X 

X 

X 

Neutral  Speed 

Threatening  Enemy  Speed 

X 

(Non-threatening)  Enemy  Speed 

X 

Sea  State 

Number  ofUSV(s) 

X 

X 

X 

X 

X 

Number  of  Contaets 

X 

X 

%  Enemies 

USV(s)  Permitted  Range  from  HVU 

X 

X 

Camera  Range 

X 

X 

%  Threatening  Enemies 

X 

X 

X 

Simulation  Length 

X 

X 

For  comparison  purposes,  we  want  to  isolate  the  impact  of  the  explanatory  factors 
common  to  all  (or  most)  of  the  analyses.  Because  the  design  matrix  is  nearly  orthogonal, 
these  contributions  are  essentially  equal  to  the  R-squared  values  for  fitting  reduced 
models.  The  three  terms,  number  of  USVs,  USV  speed,  and  camera  range  explain 
27.20%  of  the  variance  in  the  proportion  of  detection  for  the  Waypoint  scenario  and 
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16.14%  of  the  variance  for  the  Interceptor  scenario.  This  leads  us  to  say  that  if  time 
permits,  understanding  the  current  operational  scenario  and  accurately  estimating  any 
non-controllable  factors  may  help  the  decision-maker  choose  appropriate  settings  for  the 
controllable  factors. 

As  previously  mentioned  throughout  the  analyses,  a  quadratic  term  with  a 
negative  coefficient  means  there  are  diminishing  returns  and  perhaps  even  diminishing 
performance.  The  thesis  varied  the  number  of  USVs  from  1  to  24  and  the  largest  number 
of  USVs  that  would  add  to  the  mission  without  a  decreasing  effect  is  20  USVs.  This 
number  of  USVs  is  achievable  for  a  Carrier  Strike  Group  (CSG)  or  Expeditionary  Strike 
Group  (ESG)  that  is  comprised  of  several  ships. 

Although  not  impossible,  procuring  a  fleet  of  USVs  is  not  an  overnight  process. 
A  look  at  what  outcome  can  be  predicted  by  one  and  four  USVs  for  each  MOE  would 
likely  be  more  useful  in  the  near  future  than  looking  at  the  big  picture.  To  accomplish 
this,  we  set  up  a  “current”  case  intended  to  reflect  the  current  position  where  only  one 
USV  is  available,  a  “near-term”  case  where  there  are  four  USVs  available,  and  compare 
these  to  situations  where  the  number  of  available  USVs  is  not  constrained. 

Eor  the  Waypoint  model,  the  settings  for  the  other  significant  factors  are  USV  and 
enemy  speeds  set  at  20.1  knots,  the  number  of  contacts  at  49  (a  plausible  number),  the 
camera  range  at  eight  nautical  miles  (the  range  for  the  current  configuration),  and  the 
time  on  station  at  four  hours.  With  one  USV  the  prediction  interval  is  (0.2556,  0.3608), 
with  four  USVs  the  interval  is  (0.3495,  0.4303),  and  the  interval  is  (0.5291,  0.6005)  when 
the  number  of  USVs  is  set  to  17.4  (its  apparent  “best”  value).  Note  that  the  interval  is 
widest  when  there  is  one  USV  and  narrowest  when  there  are  17.  This  means  that  not 
only  does  the  average  detection  probability  improve  as  the  number  of  USVs  increases, 
but  also  that  the  results  are  slightly  more  consistent. 

Eor  the  Interceptor  model,  the  current  case  for  comparison  is  to  set  the  speeds  to 
20.1  knots,  the  combat  radius  is  five  nautical  miles  (the  current  capability),  the  camera 
range  to  eight  nautical  miles  (the  current  configuration),  and  a  time  on  station  of  four 
hours.  With  one  USV  the  prediction  interval  is  (0.1056,  0.2342),  with  four  USVs  the 
prediction  interval  is  (0.1970,  0.2864),  and  the  interval  is  (0.3705,  0.4511)  with  19.9 
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USVs  (its  apparent  “best”  value).  As  before,  the  width  of  the  interval  deereases  as  the 
number  of  USVs  inereases,  so  adding  USVs  improves  both  the  mean  and  the  eonsisteney 
of  the  MOE. 

For  regression  trees,  eomparisons  ean  be  made  using  the  leaf  eharaeteristies,  as 
we  now  illustrate.  In  the  proportion  analyses,  the  first  split  oeeurs  at  three  USVs  for  both 
MOEs,  so  the  lower  partition  is  used  for  one  USV  and  the  upper  partition  is  used  for  four 
USVs.  Here,  if  the  threat  density  among  all  eontaets  is  less  than  17%,  there  is  no 
dependenee  on  the  number  of  USVs  and  the  mean  overall  proportion  of  deteetions  is 
0.4409  (eomputed  from  Table  7).  If  the  pereentage  of  threatening  enemies  is  at  least  17% 
and  the  USV  speed  is  greater  than  28  knots,  there  is  no  dependenee  on  the  number  of 
USVs  and  the  mean  overall  proportion  of  deteetions  is  0.7955  (eomputed  from  Table  7). 
Inereasing  the  number  of  USVs  from  one  to  four  is  benefieial  when  the  pereent  of 
threatening  enemies  is  at  least  17%  and  the  USV  Speed  is  less  than  28  knots;  in  this  ease, 
the  mean  overall  proportion  inereases  from  0.3652  to  0.7035. 

For  the  proportion  of  threatening  enemies  deteeted,  if  there  is  less  than  12%,  the 
mean  proportion  of  deteetions  is  0.8000.  When  the  proportion  is  greater  than  or  equal  to 
12%  the  dependenee  on  the  number  of  USVs  is  seen.  If  a  single  USV  is  available  and  the 
pereentage  of  threatening  enemies  is  greater  than  12%,  then  the  mean  is  0.9485  (the 
weighted  average  of  the  means  in  the  2“‘*  and  3'^‘*  rows  in  Table  8).  If  four  USVs  are 
available  and  the  pereentage  of  Threatening  Enemies  is  greater  than  12%,  then  the  mean 
of  the  proportion  of  deteetions  is  0.9995. 

For  the  third  MOE  (the  number  of  threatening  enemies  that  reaeh  the  HVU)  a  split 
oeeurs  at  four  USVs  (Table  9).  Therefore,  there  is  still  a  separate  predietion  for  one  and 
for  four  USVs.  With  one  USV,  the  mean  number  of  threatening  enemies  that  reaeh  the 
HVU  is  8.5044.  Having  four  USVs  reduees  the  number  of  threatening  enemies  that  reaeh 
the  HVU  to  1.075.  If  the  number  is  greater  than  four  and  less  than  11,  the  number 
inereases  to  2.1070  threatening  enemies  that  reaeh  the  HVU. 
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V.  CONCLUSIONS 


The  primary  goal  of  this  thesis  is  to  eome  to  the  aid  of  the  Navy  and  state  whether 
and  how  the  USV  should  be  ineorporated  into  maritime  missions.  This  thesis  eannot 
provide  definitive  answers,  but  it  does  present  some  useful  and  interesting  initial  findings. 
As  pointed  out  previously,  simulation  is  an  abstraetion  from  reality,  and  therefore  the 
results  should  be  viewed  as  insights  rather  than  speeifie  numerieal  values.  For  example, 
reported  MOE  values  should  be  used  in  a  relative  eomparison  to  the  other  values. 
Regardless,  there  is  no  other  stoehastie  simulation  of  USV  taeties  known  to  have  been 
eondueted  before  this. 

Three  seenarios  were  eonstrueted  and  analyzed:  a  Waypoint  seenario  and 
Intereeptor  seenario  for  intelligenee,  surveillanee  and  reeonnaissanee  seenarios,  along 
with  a  Foree  Proteetion  seenario.  A  design  of  experiments  (DOE)  approaeh  was  used  to 
extraet  data  from  these  seenarios  for  analysis  purposes.  The  resulting  models  show  the 
relationship  between  the  proportion  of  deteetions  or  number  of  enemy  agents  that  reaeh 
the  high  value  unit  (HVU)  and  the  faetors  (10  for  ISR  seenarios  and  11  for  the  FP 
seenario)  that  are  varied  throughout  the  design.  The  breadth  of  the  thesis  study  is  to 
expand  the  eurrent  limitations  of  the  Spartan  Seout  to  eheek  for  thresholds,  relationships 
and  where  emphasis  should  be  plaeed  on  the  aetual  expansion  of  the  signifieant 
eapabilities. 

The  faetors  that  are  in  the  DOE  are  the  speed  of  eaeh  type  of  agent,  the  sea  state, 
the  number  of  USVs,  the  total  number  of  eontaets  and  the  pereentage  of  eontaets  that  are 
enemies,  the  range  of  the  eamera  sensor,  the  distanee  that  eaeh  USV  is  permitted  to  travel 
away  from  the  HVU,  and  the  USV  time  on  station.  Some  eharaeteristies  are  kept 
eonstant  throughout  the  experiment,  sueh  as  the  probability  of  deteetions  for  all  sensors 
and  the  probability  of  kill  for  the  weapon.  The  sensors  are  also  the  same  for  eaeh  elass  of 
agent.  Finally,  faetors  that  are  not  ineluded  but  eould  be  ineorporated  in  future  models 
inelude  lateney,  temperature  inversions,  eurrents,  and  other  meteorology  and 
oeeanography  features. 
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There  are  three  types  of  faetors  that  are  varied  in  the  DOE.  First  are  the 
uneontrollable  faetors.  These  are  the  speeds  of  the  non-friendly  agents,  the  total  number 
of  contaets,  pereentage  of  eontaets  that  are  enemy  eontaets,  and  the  sea  state.  Next  are 
the  faetors  that  need  to  be  eonfigured  into  the  design  and  tasking  of  the  USV;  the  number 
available  to  deploy,  the  eamera  range,  and  the  range  that  the  US  Vs  are  permitted  to  travel 
away  from  the  HVU.  One  last  pair  of  faetors  is  also  eontrollable  in  the  sense  that  a  time- 
dependent  situation  will  deeide  their  values.  These  faetors  are  the  speed  of  the  USV  and 
the  number  to  deploy  in  the  given  situation. 


A,  INSIGHTS  FOR  USV  DESIGN  AND  DEPLOYMENT 

The  uneontrollable  faetors  are  beyond  the  eontrol  of  friendly  forees.  If  this  group 
turned  out  to  have  the  highest  impaet  on  the  MOEs,  it  would  imply  that  the  USVs  eould 
not  be  deployed  effeetively  without  aeeurate  knowledge  or  assumptions  about  the  enemy 
position.  Only  the  regression  tree  analysis  of  the  FP  simulation  model  produeed  results 
that  eontained  uneontrollable  faetors  as  signifieant  terms. 

For  the  three  FP  MOEs  (the  proportion  of  all  enemies  deteeted,  proportion  of 
threatening  enemies  deteeted  MOEs,  and  number  of  threatening  enemies  that  reaeh  the 
HVU),  the  only  faetors  that  were  found  signifieant  were  the  pereentage  of  threatening 
enemies  and  the  threatening  enemy  speed.  The  threatening  enemy  speed  was  only  found 
signifieant  in  the  number  of  threatening  enemies  that  reaeh  the  HVU  MOE.  Even  though 
the  level  of  these  faetors  eannot  be  eontrolled,  they  should  be  eonsidered  when 
determining  appropriate  levels  for  the  eontrollable  faetors  mentioned  previously. 

The  eonfiguration  faetors  should  be  implemented  in  USV  design.  Analysis  shows 
that  the  optimal  number  of  USVs  operationally  available  per  HVU  is  in  the  range  of  16- 
20.  This  is  a  large  number  to  proeure  per  HVU,  and  may  not  even  be  feasible  beeause  of 
spaee  limitations  aboard  the  ships  and  the  need  for  extra  platforms  to  sustain  this 
availability.  However,  a  qualitative  summary  of  the  results  indieates  that  a  single  USV  is 
insuffieient,  that  adding  USVs  improves  performanee  in  all  three  models  up  to  a  point, 
but  that  the  performanee  eventually  levels  off  and  may  even  deteriorate  slightly.  The 
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need  for  a  relatively  large  number  of  USVs  to  eover  a  1600  sq-nm  area  may  indieate  that 
other  more  complex  methods  for  coordinating  USV  actions  should  be  considered. 

The  other  two  design  factors  are  the  camera  range  and  the  tactical  range  from  the 
HVU.  These  turn  up  as  interactions  in  the  Interceptor  model  with  the  endurance,  and  the 
camera  range  interacts  with  the  USV  speed.  The  main  effect  of  the  camera  range,  in  the 
Waypoint  and  Interceptor  models,  has  a  positive  coefficient  which  supports  the  general 
belief  that  more  capability  is  better.  However,  the  quadratic  effect  shows  up  in  both 
models  with  a  negative  coefficient  which  means  that  there  is  a  point  where  the 
performance  will  decrease. 

Combat  radius  is  only  significant  in  the  Interceptor  model  as  a  positive  main 
effect  and  a  negative  quadratic  effect.  The  lack  of  significance  for  the  Waypoint  model 
suggests  that  if  USVs  are  solely  tasked  to  follow  preplanned  mission  profdes,  the  Navy 
should  not  spend  money  improving  combat  radius  if  it  does  not  improve  the  operations. 
Experimentations  show  the  same  effect  for  either  factor  interacting  with  the  endurance 
and  leads  to  the  conclusion  that  increasing  the  camera  range  does  not  require  an  increase 
in  the  combat  radius.  Neither  of  these  factors  was  found  to  have  significant  interactions 
in  the  other  regression  analysis. 

Soon,  a  decision-maker  aboard  CSG  or  ESG  will  have  the  ability  to  choose  the 
number  of  USVs  to  deploy  and  the  speed  at  which  they  operate.  In  this  case,  an 
observation  of  the  situation  at  hand  should  be  conducted.  The  regression  models  or 
regression  tree  analyses  in  Chapter  IV  can  be  used  to  generate  point  estimates  of  the 
desired  MOEs.  These,  in  turn,  can  assist  the  decision-maker’s  assessment  of  alternatives. 
Interval  estimates  (such  as  those  illustrated  with  JMP™’s  profiling  tool)  can  be  obtained 
via  software  after  refitting  the  models  to  the  experimental  data  (Steele,  2004). 

Since  currently  only  one  prototype  USV  is  operable,  the  probability  of  detecting 
enemies  is  not  very  high  for  any  of  the  three  models.  Eor  the  near  term,  it  is  interesting  to 
know  the  potential  benefit  of  a  slightly  larger  number  of  USVs.  When  all  other  factors 
are  held  at  levels  typical  of  the  Spartan  Scout’s  existing  configuration,  an  increase  to  four 
USVs  leads  to  a  substantial  improvement  in  all  MOEs.  Eour  was  chosen  because  it  was  a 
relatively  small  number  but  appeared  as  a  split  in  the  regression  tree  analysis  for  the 
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Force  Protection  model.  Increasing  the  number  of  USVs  from  one  to  four  also  narrowed 
the  confidence  intervals  for  the  mean  responses. 

Overall,  one  surprising  piece  of  information  is  that  the  sea  state  is  not  significant 
in  the  performance  of  any  of  the  MOEs  analyzed.  This  could  be  a  function  of  how  sea 
state  was  modeled:  we  included  influence  on  USV  maneuverability,  but  not  the  impact  on 
degrading  sensor  performance.  It  is  also  interesting  to  note  that  the  enemy  speed  appears 
in  the  Waypoint  model  as  a  significant  main  effect  and  an  interaction,  yet  it  was  not 
present  at  all  in  the  final  Interceptor  model. 

As  mentioned  with  the  Interceptor  model,  an  increase  in  either  the  camera  range 
or  the  combat  radius  increases  the  proportion  of  detection  without  having  to  increase  the 
other  factor.  Similar  results  might  hold  for  other  types  of  sensors.  The  results  suggest 
recommending  only  increasing  the  capacity  of  one  of  these  ranges.  The  decision  of 
which  range  should  be  increased  is  dependent  on  the  costs,  the  technological  feasibility  of 
improving  one  attribute  rather  than  the  other;  and  which  makes  logical  sense. 

Although  sea  state  was  not  a  significant  factor,  the  actual  camera  range  truly 
depends  on  the  height  of  the  waves.  This  relationship  was  not  modeled  due  to  the 
limitations  of  the  PYTHAGORAS  software.  If  the  wave  height  is  greater  than  the  height 
of  eye  of  the  camera,  or  creates  sufficient  haze,  an  increased  camera  range  would  not 
have  any  effect.  Therefore,  we  recommend  increasing  the  combat  radius  of  the  USV 
from  the  HVU.  Since  the  Spartan  Scout  is  controlled  by  radio  frequency  this  could  be 
accomplished  by  broadcasting  the  radio  frequency  via  an  airborne  relay,  instead  of  the 
ship’s  mast. 


B,  AGENT-BASED  SIMULATION  EXPERIMENTS 

Agent-based  modeling  is  a  tool  that  can  be  used  by  the  military  to  represent 
individual  entities  that  are  a  part  of  a  larger  group,  but  ultimately  have  the  same  orders. 
The  individual  entities  are  given  the  orders  but  are  each  able  to  look  at  the  current 
environmental  conditions  and  determine  the  best  movement  and  action  for  the  individual 
agent  and  is  still  within  compliance  of  the  original  orders.  The  simulation  provides  ways 
to  analyze  situations  where  actual  field  experimentation  is  either  impossible  to  do 
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thousands  of  times  or  the  risk  to  the  personnel  and  eost  of  the  equipment  is  too  high  to 
duplieate  thousands  of  times.  Utilizing  agent-based  simulation,  military  applieations  ean 
be  looked  at  without  having  to  put  military  personnel  at  risk  as  well  as  saving  potential 
equipment  and  monetary  losses. 

The  above  paragraph  is  a  moot  point  when  the  level  of  fidelity  and  resolution 
eannot  be  reaehed.  There  are  downfalls  to  using  any  time  step  simulation  platform.  One 
mentioned  previously  is  the  ehallenge  of  defining  the  length  of  a  time  step  in  the  terms  of 
the  simulation  model,  while  maintaining  a  preset  distanee  representation  and  reasonable 
speeds.  Sueh  a  restrietion  ean  eause  the  loss  of  resolution  either  in  the  distanee  a  pixel 
represents  or  in  representing  what  happens  during  the  length  of  a  single  time  step.  In 
addition  to  the  length  of  the  time  step,  modeling  the  waypoints  required  the  speed  and 
waypoint  distanees  to  be  synehronized  appropriately.  This  too  was  a  funetion  of  the  time 
step.  Diserete-event  simulation,  where  the  future  events  drive  the  time  cloek,  would 
eliminate  the  need  to  monitor  agents  loeations  and  aetions  only  at  predetermined  times. 
Instead,  the  simulation  would  keep  traek  of  when  the  next  event  will  oeeur,  move  the 
time  eloek  to  that  point,  and  update  the  agents’  states  and  positions  appropriately. 

Converting  PYTHAGORAS  to  a  diserete-event  platform  would  not  be  a  simple 
task,  but  if  diserete-event  models  similar  to  the  Waypoint,  Intereeptor,  and  Foree 
Proteetion  seenarios  eould  be  built,  it  would  be  interesting  to  eompare  results  to  those  of 
this  thesis.  If  expanded  or  new  models  of  USV  deployment  are  built  in  PYTHAGORAS 
or  a  similar  time  step  modeling  platform,  we  reeommend  thinking  earefully  about  what 
the  pixels  represent,  potential  speeds  that  will  be  used  and  the  desired  resolution  of  the 
time  step  before  developing  the  seenarios.  In  this  thesis,  we  arbitrarily  pieked  a  distanee 
to  be  represented  by  a  pixel  in  the  initial  seenario-building  proeess.  Later  on,  there  were 
very  few  options  for  ehoosing  the  time  step  representation  that  allowed  the  simulation 
models  to  work  as  desired  without  resealing  all  of  the  distanees. 

Another  reeommendation  is  to  make  the  agents  ehange  their  behaviors  and/or 
properties  using  some  meehanism  other  than  triggers  aetivated  by  eolor  changes,  since 
the  color  changes  also  determine  whether  agents  perceive  other  agents  as  friendlies, 
neutrals,  or  enemies.  The  most  prominent  illustration  is  in  Scenario-FP. 
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PYTHAGORAS  is  set  up  so  that  the  different  agents  in  a  elass  have  speeds  within  a 
defined  toleranee,  but  eaeh  individual  USV  does  not  ehange  speed  if  even  if  it  deteets  and 
begins  ehasing  a  threatening  enemy.  Adding  the  ability  for  the  agents  to  ehange  speeds 
when  in  pursuit  of  an  enemy  would  make  the  model  more  like  the  taetieal  seenario  of 
Foree  Proteetion. 

Despite  these  eaveats,  the  DOE  approaeh  was  extremely  valuable  for  exploring 
the  seenarios  and  uneovering  insights.  Future  studies  should  eontinue  to  examine  many 
faetors  simultaneously.  Using  stepwise  regression  simplifies  the  proeess  of  determining 
whieh  faetors  are  not  important,  espeeially  sinee  the  full  models  eontain  65  or  77  faetors. 
This  automated  proeedure  did  not  do  all  of  the  work  and  is  not  a  substitute  for  human 
judgment:  the  results  still  needed  manual  tweaking  to  aehieve  a  solid  model  with  a 
simplified  list  of  terms. 

Regression  trees  are  very  useful  when  linear  regression  does  not  provide  adequate 
models.  This  non-parametrie  analysis  was  able  to  give  insights  for  the  MOEs  with  only  a 
few  faetors  in  a  eomprehensible  manner.  Other  insights  to  draw  from  the  thesis:  although 
regression  does  a  good  job  of  making  predietion  equations,  it  eannot  be  done  on  all  sets 
of  data.  Sometimes  other  means  are  neeessary,  sueh  as  regression  trees,  to  find 
relationships  between  the  faetors  varied  in  the  DOE  and  the  outeomes. 


C.  RECOMMENDATIONS  FOR  FUTURE  WORK 

An  exploratory  investigation  inevitably  leads  to  more  questions.  With  simulation 
as  the  medium  for  generating  data,  simplifying  assumptions  must  be  made.  This,  in 
itself,  is  not  a  negative  eomment  beeause  the  purpose  of  modeling  is  to  extraet  the 
essential  eharaeteristies  of  a  system.  For  this  thesis,  assumptions  were  made  on  eertain 
aspeets  of  the  USV,  its  operations,  and  its  environment.  These  are  deseribed  in  earlier 
chapters  so  the  reader  can  understand  what  has  and  has  not  been  included  in  the 
modeling.  However,  some  of  these  could  be  looked  at  more  in  depth  to  see  if  relaxing 
assumptions  provides  a  more  accurate  representation  of  real-world  operations. 
Additionally,  some  other  aspects  of  the  FP  scenario  merit  further  research.  Brief 
descriptions  follow. 
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1,  Analysis  with  METOC  Factors  Included 

The  meteorological  factors  omitted  from  the  scenarios  are  the  wind,  current,  tides, 
and  sea  temperatures.  Based  on  the  Joint  Meteorology  and  Oceanography  (METOC) 
Handbook  and  the  USSOC  Manual  Number  525-6,  factors  other  than  wave  height  need  to 
be  analyzed  in  order  to  keep  the  crew  and  equipment  as  safe  as  possible.  The  analysis  in 
this  thesis  only  considers  the  wave  height  since  we  believed  a  priori  that  this  would  have 
the  largest  impact.  Designing  an  experiment  that  includes  a  more  detailed  meteorological 
representation,  as  well  as  the  factors  used  in  the  current  analysis,  could  yield  a  more 
accurate  sense  of  how  the  USV  will  behave  in  the  open  seas. 

2,  High  Sea  States 

US  Vs  are  a  very  new  technology,  and  this  thesis  is  the  first  time  PYTHAGORAS 
has  been  used  for  USV  support.  There  currently  is  not  a  precise,  or  even  ballpark,  way  to 
determine  if  the  scenario  is  accurately  defining  the  situation  in  high  seas,  due  to  lack  of 
tactical  data.  This  influenced  our  decision  to  set  up  scenarios  only  for  the  USV’s 
“favorable”  operating  conditions  (sea  states  1,  2,  and  3).  Wave  heights  of  6-8  feet  are 
“marginal”  operating  conditions  for  USVs.  This  range  is  above  sea  state  3,  and  the  USV 
performance  behavior  is  likely  to  be  quite  different  when  it  is  operating  in  marginal 
conditions  rather  than  operating  in  favorable  ones  (Joint  METOC  Handbook,  2000). 

As  USVs  are  given  more  opportunity  to  be  tested  and  utilized,  data  could  be 
collected  under  high  sea  state  operations.  This  type  of  information  could,  in  turn,  be  used 
to  build  and  perhaps  even  calibrate  models  for  USV  operations  in  high  sea  states.  The 
approach  described  in  this  thesis  provides  a  template  for  this  type  of  analysis,  and 
combining  the  results  could  very  well  give  a  good  overall  view  of  the  conditions  in  which 
the  USV  can  operate  effectively. 

3,  Rescale  Simulation  Model 

As  mentioned  previously,  latency  was  not  included  due  to  the  time  step  length.  A 
change  to  the  scale  of  the  model,  allowing  the  time  step  to  be  smaller,  would  permit  the 
agents’  behaviors  between  the  current  72-second  time  steps  to  be  drawn  out  of  the 
simulation.  To  change  the  scale,  the  pixel  definition  and  the  scale  of  the  speed  have  to  be 
altered.  The  time  step  would  have  to  be  changed  so  that  the  fraction  of  a  second  needed 
for  the  host  ship  to  record  information  from  the  USV  is  noticed.  Changing  this  feature  in 
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the  modeling  might  give  more  aeeurate  answers  to  the  questions  presently  posed  in  the 
eurrent  analysis,  although  at  the  eost  of  a  larger  eomputing  effort. 

Along  with  ehanging  the  aetual  seale  of  the  models,  it  would  be  useful  to  have  a 
eloser  look  within  the  range  of  faetors  that  are  varied.  For  example,  the  number  of 
eontacts  was  extremely  exaggerated  so  potential  threshold  values  were  not  missed.  Sinee 
there  are  thirty-three  levels  on  a  range  of  1-500,  different  NOLH  designs  involving  more 
faetor  levels  eould  be  used  so  that  greater  fidelity  is  seen  through  out  the  range.  With  500 
eontaets  in  a  1600  sq-nm  area,  the  maximum  eontaet  density  is  one  contaet  per 
approximately  three  square  nautieal  miles.  Greater  eontrol  over  models  would  support 
higher  eontaet  densities  in  more  foeused  area  of  operations. 

4,  The  Effect  of  Threatening  Enemies  Reaching  the  HVU  in  the  Force 
Protection  scenario 

Taking  a  look  at  how  the  US  Vs  reaet  when  a  hostile  enemy  reaehes  the  HVU 
eould  be  the  next  step  into  determining  the  purpose  of  the  USV  when  a  threat  is 
imminent.  One  view  is  that  the  HVU  would  have  mueh  more  on  its  hands  than  worrying 
about  what  the  USVs  are  doing  at  this  point  sinee  it  is  known  that  the  personnel  of  the 
HVU  are  going  to  be  in  grave  danger  when  a  threatening  eontaet  is  under  the  nose  of  the 
ship.  However,  there  may  still  be  a  need  for  the  USVs  onee  the  threatening  eontaets 
reaeh  a  eertain  radius  within  the  HVU.  This  could  be  modeled  in  PYTHAGORAS,  or 
some  other  agent-based  platform,  but  the  scenario  should  represent  the  attack  on  the 
HVU,  the  USVs  reactions,  and  the  aftermath.  Such  a  scenario  would  probably  involve  a 
more  detailed  representation  of  the  HVU,  such  as  allowing  the  HVU  to  move. 


D,  SUMMARY 

Preventing  fatal  incidents  such  as  the  April  2004  maritime  interdiction  operations 
occurrence  is  an  advantage  to  implementation  of  the  USV  into  Naval  missions.  In  this 
thesis,  multiple  linear  regression  and  regression  trees  are  coupled  with  a  DOE  that 
analyzes  up  to  11  factors  simultaneously.  These  illustrate  how  the  USV  can  be 
introduced  into  the  Fleet  to  effectively  assist  in  ISR  and  FP  activities.  The  results  provide 
several  operational  and  tactical  insights,  and  form  the  basis  for  a  recommendation  to  the 
US  Navy  to  use  the  USV  in  an  active  role  in  maritime  missions.  They  also  provide 
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guidance  on  the  benefits  of  improving  USV  sensing  and  enduranee  capabilities,  and 
reveal  that  simply  maximizing  numbers  of  US  Vs  is  not  neeessary  for  attaining  high 
mission  performanee. 
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LIST  OF  ACRONYMS 


Cl 

Confidence  Interval 

DOE 

Design  of  Experiments 

FP 

Force  Protection 

GET 

USS  GETTYSBURG 

HVU 

High  Value  Unit 

ISR 

Intelligence,  Surveillance,  and  Reconnaissance 

MIO 

Maritime  Interdiction  Operations 

RE 

Radio  Frequency 

RHIB 

Rigid  Hull  Inflatable  Boat 

USV 

Unmanned  Surface  Vehicles 
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